Compare Programs and Program Components

The preceding chapters explained how and why to collect information on costs, procedures, processes, and outcomes (effectiveness and monetary benefits) of substance abuse treatment programs. This chapter provides strategies for deciding on changes in program operations that should improve program cost-effectiveness and cost-benefit. They are rooted in a common sense (and quantitative) approach to managing treatment programs called operations research. These steps can be accomplished with simple graphs or complex mathematical equations. (See Yates 1980, 1996, for more complex examples and more mathematical strategies.) As with most endeavors, the more effort and time you devote to cost-effectiveness and cost-benefit, the greater the potential rewards.

Cost-Outcome Decisions

Sometimes cost-outcome analysis is simple. If the question is "Which of these two programs should be funded?" a quick decision may be possible. If, for example, Program A has much better outcomes than Program B, and Program A clearly costs much less than Program B, the decision is clear-cut. Program A is more effective or more beneficial (or both) and is less costly. Table 26 presents (a) the ways in which two programs can differ or be similar to each other in outcomes and costs and (b) the cost-outcome decisions that result.

The same simple decision rules can be applied to two different treatment procedures or even two therapists. These rules are summarized in table 26, which is called a Fishman table in honor of the researcher who first applied this table to cost-outcome analysis.

Table 26. Cost-Outcome Decision: Program A or Program B?

However, the simple phrases "Program A costs less than Program B" and "Program A has better outcomes than Program B" hide a dilemma: How does one decide when one program costs less than another, or when one program has better outcomes than another?

Once adjustments have been made for differences in the number of patients served in competing programs, statistical analyses can answer the question of whether a difference in costs or outcomes is real and not just due to chance variations in the cost or outcome numbers that were generated by the competing programs. Most spreadsheets come with add-in programs or macros that perform t tests and several other statistical tests. You have to show which columns of numbers (e.g., outcome data for Programs A and B) you want to compare to see if the apparent difference is real.

Statistical tests do not, however, answer how big or how important a difference is. Statistical tests tell whether an apparent difference is not just due to chance. The size of a difference can be described with average costs and other numbers. The importance of a difference is a judgment that can be made by surveying community and patient representatives.

The Fishman table also illustrates another problem with simple comparisons of outcomes and costs. Even if there are only two programs, a Fishman table does not indicate which decision is correct if Program A has better outcomes than Program B but also costs more than Program B.

Cost-Benefit Ratios

Even if Program A's benefits exceed its costs, the question of which should be funded still is not answered. It is possible that Program B's benefits exceed its costs as well. Should both B and A be funded? Maybe, but only if their net benefits are similar. You could compare the net benefit of Programs A and B to see which is bigger, and then choose the program that has the bigger net benefit.

Check that the bigger net benefit of, say, Program B is not just a result of Program B serving more patients at the same level of effectiveness as Program A. Serving more patients can result in a bigger benefit, and a bigger cost and a bigger net benefit, without the program being any better than a smaller program.

Suppose Programs A and B both double the value of cost -they both generate the same ratio of benefits-to-costs of 2 to 1. Suppose, too, that Program A gets funded at $100,000 a year to serve 100 patients from its district of 100,000 people while Program B gets funded at $500,000 a year to serve 500 patients from district of 500,000 people. (Note that Program A and Program B have the same rates of being funded at $1,000 per patient. They also draw patients at the same rate of 1 per 1,000 persons residing in their districts.)

Reflecting the programs' identical benefit/cost ratios, Program A produces benefits that are double its cost for a net benefit of $100,000. Program B produces benefits that are double its cost for a net benefit of $500,000. Which program is better? Neither. Program B just has a bigger funding base and thus appears better. Its performance is no better or worse than Program A's. That's easy to see if the net benefit per patient is calculated. Dividing the net benefit for Program A results in the same net benefit per patient as for Program B:

$100,000 net benefit ÷ 100 patients = $1,000 net benefit per patient.

$500,000 net benefit ÷ 500 patients = $1,000 net benefit per patient.

Cost-Effectiveness Ratios

Problems due to large differences in funding base or patient load are less likely to distort cost-outcome analyses that use nonmonetary measures of outcome. When the effectiveness of a program is measured, it usually is measured for each patient individually. Effectiveness measures usually retain their per patient units, as in average drug-free days per patient. Once the cost of treating a patient is figured out, it is easy to divide the cost of treating the patient by the effectiveness measure to arrive at a ratio of cost to effectiveness. These cost-effectiveness ratios can be calculated for each patient in a program and averaged to describe the typical cost-effectiveness of treatment.

Cost-effectiveness ratios are most useful in decisionmaking when compared to cost-effectiveness ratios for other programs. If Program A requires an average $27.43 per drug-free day produced, whereas Program B requires an average $30.71 per drug-free day produced, then Program A appears to be preferable because it is more cost-effective than Program B -at least on this one measure of effectiveness.

Cost per drug-free day is a cost-effectiveness ratio that has a number of useful characteristics. A day free of drugs is something concrete that most people understand. Many people can appreciate a day free of drugs as a challenging endeavor and an important achievement. Because of this, and because of the concrete value of money, the cost of producing this day free of drugs also becomes more tangible.

Cost per drug-free day suggests a standard metric that will be better (lower cost per drug-free day) if either (a) less money is spent per patient, (b) more patients are free of drugs for a day, or (c) an individual patient is free of drugs for more days. The problem with this cost-effectiveness ratio is that you do not know whether condition (a), (b), or (c) has occurred. In truth, probably each of these three conditions occurred for different patients within the program.

Cost-Effectiveness Ratios Versus Cost-Benefit Ratios

You might ask yourself "How could Program A be more cost-effective than Program B, when they were equal in terms of cost-benefit? Because benefit measures usually are derived from effectiveness measures by multiplying the effectiveness data by a monetary amount, shouldn't cost-effectiveness and cost-benefit be the same? In fact, why do we need both? Why not just use the cost-benefit findings?"

Program A could be more cost-effective than Program B according to one measure of effectiveness (patient drug-free days) because Program A did a better job of changing patient behaviors related to that measure of effectiveness. Or, maybe Program A used less costly procedures to change the patient behavior. Program A might also be less cost-effective than Program B on another measure of effectiveness (e.g., employment of patients). The superiority of Program A in terms of the monetary value of more drug-free days might be canceled out by the superiority of Program B in terms of the monetary value of more employment for patients. So, the overall cost-benefit of Programs A and B could be the same even though they differed on specific measures of effectiveness.

This discussion points out an advantage of cost-benefit analysis, whether done with ratios or net benefit calculations: Cost-benefit analysis gives a single answer (as long as one measure of cost and one measure of benefit are used). That makes decisions easier. Cost-effectiveness analysis, whether done with ratios or other methods, shows how programs differ on specific measures of outcome. This is better if you want to focus on one or two measures, or if you do not agree with the way that placing a money value on an effectiveness measure biases the overall evaluation toward one measure or another.

When Cost-Effectiveness and Cost-Benefit Converge

Cost-effectiveness and cost-benefit analyses can both generate overall measures that describe how well a program uses its resources to achieve its goals. Calculating the cost-effectiveness index seems like a lot of work, but, if you'll recall the calculations needed to turn effectiveness measures into monetary benefit measures, there was a lot of work in the cost-benefit analysis, too. The benefit calculations weighted for importance each effectiveness measure in terms of money saved or money produced. The calculations involved in generating the overall effectiveness index substituted the importance weightings for the different monetary rates used in the benefits calculations.

You may prefer to use the importance weightings inherent in the different monetary values assigned to a "drug-free day" or "employed days." These values will change over time as the value of money changes through inflation, deflation, or currency adjustments. Also, as the market changes, the value of a drug-free day and the amount one is paid for a day of employment will change. Actual measures of program effectiveness, such as drug-free days and days employed, seem more constant. Their importance also may change over time, however, as a result of changes in community norms.

The best advice probably is to conduct a cost-effectiveness analysis and a cost-benefit analysis for each measure and each patient as well as for all measures and all patients. That way, you'll have answers to a host of questions. You'll also have more opportunities to find solutions to the problems of how to get even better outcomes out of your programs.

When Net Benefit and Ratios Fail

Net benefit and ratios of benefit over cost and of cost over effectiveness are informative because they reduce information on two sets of variables -costs and outcomes -to a single cost-outcome number. That advantage can be a disadvantage, too: When information is reduced, the cost-outcome number is more readily understood, but its context and limitations are more readily ignored.

Consider the situation in which Program A has better outcomes than Program B but also costs more than Program B. In this context, a decision to prefer the program with the higher net benefit per patient, or the higher ratios of benefit divided by cost or cost divided by effectiveness, could be incorrect for several reasons.

It is possible that the higher cost of Program A is too high. The net benefit and the ratios provide no information on budget limits. Worse yet, they do not tell you how much the program costs. That information was discarded when costs were either subtracted from benefits, or divided into benefits, or divided by effectiveness.

It is also possible that the poorer outcomes of Program B are too poor to meet minimum criteria. Funding policy, community standards, or law may dictate a certain minimum level of effectiveness or benefit, below which a program should be closed. Information about minimal effectiveness or benefits is not included in the net benefit or ratio calculations. And, as was the case with cost, information about the actual benefits and effectiveness of Program B has been discarded, replaced again by a difference or a ratio.

Cost-outcome ratios and net benefit also can obscure effects that level of funding (and size of patient load) can have on the relationship between costs and outcomes. These relationships are easier to see if information about costs and outcomes is preserved. One way to do this is with tables contrasting costs for different outcomes. Another way is with graphs.

Cost-Outcome Graphs

A decision about Programs A and B is possible, and collecting data on costs and outcomes is an important step toward making the decision. You need more information about acceptable costs and outcomes. In particular, you need to know "where the line is" -literally, as you'll see in a moment -on both costs and outcomes. Making cost-outcome decisions usually requires knowing the budget limit on costs. Some cost-outcome decisions also require knowing what basic outcomes must be achieved, at a minimum, by a program if it is to be funded. These decisions may not require all the mathematical machinations described in the preceding pages. Instead of calculating information on costs and outcomes, graph them. Graphing outcomes on the vertical axis and costs on the horizontal axis preserves information on both costs and outcomes while also helping you see how the two may be related.

In the following graphs, the values on the outcome measure would be exactly what is observed by researchers. Change, if it is to be represented on the graph, can be shown as two dots labeled "pre" and "post" for the same program, connected by a line to show their association. By doing this, differences between different programs in "pre" values are made explicit. Graphing the difference score could hide serious pretreatment differences in severity of substance abuse between programs.

Add Limits

Knowing the maximum tolerable cost (the budget limit) and the minimum tolerable outcome (minimal outcome criteria) can help in decisionmaking when the Fishman table fails to identify clearly which program (or procedure or therapist) should be chosen. Graphs of outcomes against costs help illustrate this point. Consider, for example, the following cost-outcome situations.

In graph 1, Program A clearly is the better of the two: It has a better outcome, and it costs less. The letter "A" is higher than "B" on the vertical axis, showing how good the outcomes were. The letter "A" also is to the left of "B" on the horizontal axis, showing how costly the programs were.

Likewise, in graph 2, Program A clearly is the better of the two: It has better outcomes, and it costs the same as Program B.

Graph 1. A is less costly and more effective or beneficial than B	Graph 2. For the same cost, A is more effective or beneficial than B

Graph 3 poses a problem: A has better outcomes than B, but A also costs more than B. Is A worth it? Should A be chosen and funded rather than B? Only information about budget limits and outcome criteria will answer the question. The answer even may be that neither Program A nor Program B should be funded. Both might exceed budget limits, as shown in graph 4.

Graph 3. A is more effective or beneficial , but also more costly than B	Graph 4. Budget limits in a cost-outcome graph

Both Programs A and B might not achieve minimum levels of outcome, as shown in graph 5, which also would recommend choosing neither program. A more likely scenario is that Program A exceeds the minimum acceptable outcomes, but at unacceptable cost, whereas Program B keeps costs below the budget limit but does not achieve minimum acceptable outcomes. This situation is depicted in graph 6.

Graph 5. Minimum outcome criteria in a cost-outcome graph	Graph 6. Both A and B are infeasible

Note that Program A could have benefits that exceed costs, as could Program B, but neither might meet the budget and outcome criteria. It is not necessary to increase cost maxima or lower outcome minima. What is needed, and what is likely possible, is a new program that has adequate outcomes at tolerable costs. This is Program C, shown in graph 7. Program C has a cost-outcome relationship that is positioned below the maximum cost and above the minimum outcome.

Graph 7. Only C is feasible

More Than Two Programs

Sometimes cost-outcome analysis is not simple. If a decisionmaker is offered the sort of Program A versus Program B choice described earlier, it is possible that someone has limited the choice so A is always chosen. Usually, alternative programs are possible, if not currently in operation. Often, there are more than two procedures or therapists. As the number of programs, procedures, and therapists increases, the likelihood that one program, procedure, or therapist will be the most effective or beneficial, and the least costly of them all, decreases. The decision becomes one of tradeoffs: At what point is better drug treatment not worth the additional cost? Graphs of cost-outcome relationships are helpful in these situations.

Assumptions Encouraged by Cost-Outcome Ratios and Revealed by Cost-Outcome Graphs

Ratios of cost to outcome provide a succinct index of program performance that obviously includes information on outcomes and cost. By dividing cost by effectiveness (or benefit by cost), though, potentially valuable information is lost on the actual amount of resources consumed and the amount of outcome achieved. Suppose, for example, that a program generates an average 500 drug-free days per patient for $5,000 per patient. The resulting ratio is $10 per drug-free day per patient. This ratio encourages an assumption that if more money could be spent, more drug-free days would be produced.

This is not necessarily the case. It is more likely that investing more money in treatment of each patient (say, doubling expenditures to $10,000 per patient) would result in an increase in drug-free days, but less than a doubling to 1,000 drug-free days.

It is possible, too, that doubling the funding of a program might allow it to see double the number of patients. Suppose that the cost-outcome ratio shows the cost per successfully treated patient. It is possible that the same level of effective treatment would be provided, doubling the number of successfully treated patients. One factor that works against this is the limited flexibility of many human service systems, including drug treatment systems. There simply might not be enough space and counselors to see double the number of patients. Of course, additional space could be rented and more counselors could be hired. Administrative costs would have to increase as well, in light of the increased resources being devoted to treatment.

If there is enough extra space in the program facility, and if program administrators have extra time, then double the number of patients can be seen at even less than double the cost. It is more likely, however, that limitations in the flexibility of program resources will increase the cost of adding each additional patient (sometimes called the marginal cost). This means that the return (in terms of number of successfully treated patients) on investment in treatment diminishes as more patients are added -the classic diminishing returns on investment.

Cost-outcome ratios also encourage the belief that a decrease in program funding would decrease the number of successfully treated patients by the amount indicated by the ratio. If a program's funding is decreased by 20 percent, for example, from $100,000 to $80,000 per quarter, one should not assume that outcome also will decline by 20 percent. Some programs experiencing a 20-percent budget cut might well survive and produce outcomes that decline only 5 to 15 percent. The programs might find more effective procedures (such as more group therapy and less individual therapy). Other programs might have to close their doors if their budgets are reduced by 20 percent, creating a rather sharp decline in the number of successfully treated patients.

In sum, ratios encourage funders of treatment to assume that there is a straight-line or linear relationship between costs and outcomes. The ratio is, essentially, the slope on a graph of costs and outcomes.

The preceding examples argue that it is rare to find a straight-line relationship between costs and outcomes that lasts for a significant range of costs or outcomes. This observation recommends that a better understanding of possible cost-outcome relationships could be gained by graphing costs against outcomes for a variety of programs (or program funding levels).

If you want to go beyond graphs, the next major step in understanding and improving cost-effectiveness and cost-benefit is delving into the more mathematical techniques of linear programming and other forms of quantitative operations research. As detailed in a book by Yates (1980), operations research involves the construction and solution of equations that express mathematically the relationships among costs, procedures, processes, and outcomes. Budget constraints and outcome goals are included in the mathematical expressions.

The quantitative model of the treatment system that is constructed with these equations can be solved using linear programming either to maximize outcomes that can be achieved within budget (cost) constraints, or minimize the costs of achieving set levels of outcome. Operations research provides a variety of models and solution procedures that are potentially useful for many problems facing substance abuse services.

[Manual Index] [Previous Section] [Next Section - Improving Programs...]