This chapter is based in part on Costs and Benefits of Combining Probation and Substance Abuse Treatment by Alemi F, Taxman F, Vang J, Thanner M, Baghi H. (in review)
This chapter demonstrates the application of decision trees to analysis of cost and effectiveness of clinics. Health care managers often start clinics and programs. They need to understand how much to charge for the services offered through their new investment. They need to justify the effectiveness of clinics to various funding agencies. They need to identify weak and money losing operations and work on improving them. Unfortunately, the cost effectiveness of clinics is not always clear as it is difficult to distinguish program costs from other expenditures in the organization. Many clinicians have multiple roles within the organization and it is not clear how much their time or effort is going into the new program versus other activities within the organization. It is difficult to understand how personnel in management and other indirect costs affect the viability of the new clinic. To make things more confusing, new clinics share facilities with existing programs making it difficult to charge them rent or allocate portion of capital expenditures to these clinics. A rational approach would require us to isolate the cost of the program from other costs and to decide how much of the overhead should be carried by the program.
In addition, program costs are just part of the picture. Any program or clinic not only has its own services but also affects services offered by other units. Offering a new program in cardiology might help identify patients for the existing home health care service of the organization. If the new clinic is more modern and advance than existing operations (e.g. it has an electronic medical record or it uses new surgical equipment), then it might change the image of the entire organization and affect the referral to all existing clinics. The new clinic might be the loss leader that attracts patients to other parts of the organization. In short, any new investment may have unforeseen or planned consequences. If we are evaluating the cost and effectiveness of a new clinic, it is important to go beyond program operations and costs and look at its impact on other services too. This chapter shows how healthcare managers can isolate the costs of a clinic/program and trace the consequences of a service through use of decision analytic tools.
Managers who have to justify the operations of a clinic to outside planning offices or insurance companies might be interested to look at the consequences of the clinic broadly. Instead of just looking at how the clinic affects their existing services, they might want to look at the impact of the clinic on payers. They might not want to limit the analysis to specific internal components but also include utilization of services outside of the organization. In this chapter we show how such a broad analysis of consequences of opening a clinic can be carried out.
To demonstrate our ideas, we show how we applied them to evaluation of the cost effectiveness of co-locating a substance abuse treatment clinic within a probation agency. Over 4 million adults are under probation or parole supervision. Supervision failures, often linked to return to drug use, create a cycle back into prison and jails. If supervision can be enhanced with treatment, then perhaps the cycle can be broken and offenders can return to community, not only restoring their lives but also perhaps saving money associated with crime and return to prison. In recent years there have been a number of studies of cost effectiveness of substance abuse treatment,[i]-[ii][iii][iv][v][vi][vii][viii][ix][x][xi][xii] some of these studies have focused on cost effectiveness of programs for reducing recidivism.[xiii]-[xiv][xv][xvi][xvii][xviii][xix][xx][xxi] We used a decision analytic model to evaluate the cost effectiveness of co-locating substance abuse treatment clinic within a probation agency. For ease of reference, we refer to the clinic coordinated with the probation agency as the seamless probation and compare it to the traditional model of providing probation and substance abuse treatment services independently.
Decision models have been used to model cost and benefits of a wide array of services,[xxii][xxiii][xxiv][xxv][xxvi][xxvii][xxviii][xxix][xxx][xxxi][xxxii][xxxiii][xxxiv]-[xxxv] including the cost and benefits of substance abuse prevention.[xxxvi],[xxxvii],[xxxviii],[xxxix] The typical analysis (see Figure 1) starts with a decision node, contrasting joining the new clinic and the usual or standard care alternative. Then the various events within the program are indicated (e.g. treatment retention). In Figure 1 this is shown as a visit. Since the analysis is done per day, we show this as a day in which a visit has occurred. Next two costs are reported, the cost of the program and the costs of consequences of the program. The cost of program refers to the personnel, material, information system, building and other capital and operating costs of delivering the program. Of course, these costs have to be calculated per day in which there is a visit and a day in which there is no visit so that it corresponds to the unit of analysis in which probabilities are measured.
Figure 1: A Typical Decision Tree for Analysis of Cost Effectiveness of Clinics
The cost of consequences refers to subsequent costs that the patient or the organization may incur. For example, a clinic visit may lead to additional referrals or in some cases to a hospitalization. The cost of consequences is itself a separate decision tree. In Figure 1 this is shown as node “C.” This node is broken into a more detailed tree, in Figure 2. The tree shows the daily probability of various consequences and the daily cost associated with each.
Figure 2: Typical Consequences for Analysis of Cost Effectiveness of Clinics
Note that in the analysis all probabilities and costs are calculated per day. Thus we talk of daily probability of initiating treatment and daily probability of retaining the client. For consequences, we talk of daily probability of needing a service and daily cost of various services. In this fashion, a consistent unit of analysis is kept through out the analysis. In Figure 2, for example, we are assuming that after the clinic visit the patient may be hospitalized or sent to specialist. In addition, the figure has a place holder for other major events that might occur as a consequence of the clinic visit. Finally, to make the list of possible events complete, we need to also include the situation where none of the major events imagined will happen. Strictly speaking the events depicted must be mutually exclusive and exhaustive. Some events, for example visiting a specialist and being hospitalized may occur on the same day and thus may not be mutually exclusive. If this is not occurring often, the analysis depicted in Figure 2 may continue as is and be considered an approximation of the reality.
Unlike traditional cost benefit analysis, a decision tree explicitly differentiates the probability of an event from its unit costs. For example, in Figure 2 the probability of a day of hospitalization is separated from the cost of a day of hospitalization. Because unit costs of services are unlikely to change under different alternatives, a decision analytic model reduces the number of estimates needed. For example, while the new clinic is expected to affect the probabilities of hospitalization, it is not expected to affect the unit cost of a day of hospital care. Differentiating the probability of the event from its costs allows us to rely, where necessary, on national estimates of unit costs. Therefore, the method makes data collection easier and perhaps more accurate by breaking the analysis task into smaller components and using national estimates for components that do not change under various alternatives.
The decision tree in Figure 3 shows the overall structure of the analysis for seamless and traditional probation. The left section of the tree depicts the decision. Immediately after this node, the probability of receiving one day of probation or one day of treatment is shown. Like our previous analysis these indicate utilization of services. The middle section of the tree depicts treatment and probation outcomes, what we previously called consequences of clinic visits. Next, the tree assumes that probabilities for a day of homelessness, a day of unemployment, a day of foster care for their children, a day of hospitalization, and a day in prison will vary with client’s probation and treatment status. The right hand section of the tree depicts various costs incurred by payers in association with the consequences or recidivism or return to drug use. These costs include the costs of one day of probation or one day of treatment. Since the analysis relies on daily probabilities of the events, costs are also estimated per day.
In any analysis, the perspective of analysis dictates whose costs are included. If one is analyzing costs to the patient, then cost to the hospital are not included. If one is including costs to the organization only, then costs to external agencies are not included. A societal perspective may include costs incurred to caregivers and other social components of care. If the analysis is done for payers, then costs to these organizations are included and other costs are ignored. In the seamless probation study, the focus of the analysis was costs and benefits to government agencies that were expected to fund the clinic.
The impact of most social interventions is experienced over time--the entire life of the client and perhaps into other generations. If we need to follow the consequences of opening a clinic, we need to decide how far in the future we want to look. Many of the benefits occur several years later (e.g. reduced hospitalization) while the costs occur during the clinic visits. It is therefore important to define precisely the timeframe for cost benefit analysis to discount future returns to current values. For example, the benefit of seamless probation shows years later in the form of reduced crime. To capture these, a sufficiently long perspective is needed. The study on seamless probation examined the effect of the substance abuse clinic for 2-3 years after enrollment in the study. This is long enough to see the impact of short term events but not long enough to capture life time events.
This section discusses how the probabilities needed for the analysis are calculated. There are four ways to assess these probabilities (see Table 1).
The first method is to follow a large cohort of patients (preferably randomly assigned to various alternative arrangements) over time and count the frequency of the events. This method is objective, requires considerable follow-up time and access to large number of patients. For example, we might follow 100 patients in our clinic and report the frequency with which they are hospitalized.
The second method is to examine time between re-occurrences of events in the decision tree. If we can assume the daily probability of the event has a binomial distribution, then time between the occurrences of the event has a Geometric distribution and the daily probability can be calculated by number of days in between the reoccurrence:
Daily probability of event = 1 / (1 + Number of days to reoccurrence of the event)
This method is objective but requires a shorter follow-up time than the first method, especially if the probabilities being assessed are relatively small. For example, the daily probability of hospitalization might be calculated as number of days before the patient is hospitalized. If a patient is hospitalized after 80 days, then the daily probability of hospitalization is 1/81.
The third and fourth methods are same as the first two but based on experts opinions and not objective data. We might ask about days to the event or the frequency of the event. We might ask a clinician how many days before a patient is hospitalized; or we might ask an expert about the patient’s prognosis by asking them to specify the likelihood that the patient might be hospitalized within 30 days.
In the seamless probation example, we used the first method. We recruited 272 offenders with extensive criminal justice histories and randomly assigned them to seamless and traditional probation. Offenders were interviewed at baseline and at three 12 month follow up periods to examine their utilization of various services. Of the clients, 78.01% of the clients on traditional probation and 77.10% of the clients in the seamless group were available for follow-up interviews.
When assessing objective frequencies, it is important to keep in mind that the decision tree requires the calculation of daily probability of various events. These probabilities are calculated by dividing the length of various events by the total number of follow up days. When calculated in this fashion, the probability of one day of an event is affected by the duration of the event. For example, client’s length of stay in a treatment alters the probability of a day of treatment. The longer the treatment program the larger the daily probability. A client who is in treatment for one year will have a daily probability of one. A client in treatment for one month in a year will have a daily probability of 8%. As clients stop and re-start treatment, the daily probability changes. Calculating probabilities in this fashion allows us to take into account the client’s length of treatment and multiple returns to treatment. Table 2 provides average length of various events for the clients we followed and Table 3 turns these durations into daily probabilities.
Table 2 shows that the clients in the seamless group had more treatment and probation days. The consequences of these additional treatment and probation were slightly lower number of arrests but also higher frequency of technical violation, where the probation officer issued a warrant for the client to appear in front of the judge. On the balance, the clients in the seamless group spent more days in prison/jail waiting to appear in front of the judge. The clients in the seamless group also had higher rate of homelessness. Across all these variables there were considerable variation and in none of the variables the seamless group had a statistically significant difference with the traditional probation group.
Table 3 shows the probability of various consequences for each pathway in the decision tree in Figure 3. This table is calculated by dividing the length of stay reported in Table by the follow-up period of each patient. With the exception of arrest and technical violations, these probabilities reflect both the incidence of the event and the length of the event. If seamless probation was more able to reduce adverse outcomes, one would expect lower numbers in rows associated with seamless probation.
The decision tree in Figure 3 depicts treatment occurring after participation in probation; therefore, we need to show the conditional probability of treatment given probation. This is calculated by dividing the joint probability of probation and treatment by the marginal probability of being in probation. For example, using Table 4, the conditional probability of clients in seamless probation seeking treatment while they are in probation was 0.14/0.45 = 0.30. In contrast, the same conditional probability for clients who were in a traditional probation was 0.05/0.41
Daily Cost of Clinics
The daily cost of a clinic or program is calculated from dividing the total cost of a program during a year by its yearly census. Program costs include operating and fixed costs of the program during the year. Yearly census is the number of days clients were enrolled in the program during last year. This creates some counter intuitive relationships. Very expensive programs may have a low daily cost if they also have high census. Likewise, inexpensive programs may have a high daily costs if the census is low. As census increases daily costs usually go down. This section describes how to estimate both the program costs and its census.
Program costs are estimated from the organization’s budget. Typically the budget provides cost of personnel, supplies, equipment, buildings and information services. To these costs one adds the market value of donated buildings, volunteer services and unaccounted retirement costs. Economic cost of a clinic differs from accounting cost as it includes the value of assets or personal services donated to the program. For example, accounting procedures usually depreciate building costs. This distorts real market value of the asset. To correct for these inaccuracies within the budget, whenever possible the analysis should rely on lease value of major assets such as buildings, office space or information services.
When a picture of true economic costs is established, then costs are allocated to various programs within the organization. Sometimes these allocations are clear – as when the budget of the clinic is separate from other operations. Other times costs of various programs and clinics are mixed together in the same budget and an allocation scheme must be decided upon. Table 5 provides an allocation scheme for various component of the budget.
The cost within many budget categories are allocated to the program using the activity of employees by the following formula:
Cp, c = (Cb,c+Cm,c) Ep/Eb
The key for allocation of organization’s budget to program costs is determination of personnel activities. Clearly some personnel will have dual roles and it is important to ask the percent of time they allocate to various activities. This is typically done by a survey of employees. Because of the focus on employee activities, the approach described above is often called the Activity Based Costing (ABC),
In the seamless probation, two new operations were introduced: a new clinic for providing substance abuse treatment and a new way of doing probation. The study estimated daily cost of both operations. The daily cost of providing the probation was calculated from the budget of the probation agency. Costs not directly available through the accounting system were added in. Building costs were based on lease value of equivalent office space. Cost of information services provided by other State agencies, not directly on the budget of the probation agency, was added in. Table 6 shows the cost of seamless and traditional probation. The first column shows the total agency budget. The three columns to the right of it show the allocation of these costs to various probation programs (investigative reporting, seamless supervision and traditional supervision).
A similar procedure was followed for cost of the substance abuse treatment clinic. Table 7 shows the cost of outpatient treatment received by clients at the clinic. The first column shows the total budget of the clinic. This clinic has three programs and the total budget was allocated to these three programs based on personnel activities and use of services. Of particular interest is the fact that centralized management costs did not show in the budget of the clinic but needed to be added in as a portion of these services were for the clinic.
Based on Alemi F, Sullivan T. An example of
activity based costing of treatment
A key factor in estimation of cost per day of service is the estimation of number of days of enrollment, what we call program census. It is relatively simple to calculate the cost per visit; one could count the number of visits. But clients stay in a service even when they do not visit the clinic. Furthermore, the analysis needed to estimate the cost per enrollment day (i.e. the days from admission to treatment to discharge) and not per visit as daily probabilities were being used. There are three methods to estimate enrollment days. If admission and discharge dates are available, enrollment can be measured from the difference. Sometimes this data is not readily available. In the second method, the clinicians can be asked to estimate their panel size during last month. Then enrollment days are calculated as 365 times the panel size. The third method is to look at time between most recent discharge and admission dates for the clinician. All three methods are subject to errors as clinicians may overestimate their panel size and historical discharge and admission dates may not reflect recent patterns. To make a more stable estimate, the enrollment days estimated from the various methods can be averaged. In addition, the range of the estimates can be used to guide the sensitivity analysis.
Table 7 shows estimates of the enrollment days for three different programs. There is considerable variability between the estimates. In two programs the two estimates are close to each other but for the outpatient program the two estimates are considerably different.
Typically, daily cost of consequences (e.g. day of hospitalization) does not change across alternatives. The clinic and the standard care will differ in the frequency of occurrences of various consequences but not the daily cost of each occurrence. Therefore, these daily costs can be estimated from national or regional values available through the literature.
For example, the decision tree in Figure 3 separates the probability of arrest from its costs. Because cost of arrest is unlikely to change with the use of seamless probation program, national estimates were used for these costs. Table 8 shows the estimated cost of arrest and court processing in 2001 and 2004 values.
Table 9 shows the daily cost of various consequences as estimated from published national or regional data. The cost of a day of employment was the exception to the rule. This cost was estimated by the tax paid by offenders on legal income they had during probation.
The cost of various consequences can be calculated by multiplying the daily probability of the consequence by its daily costs. Table 10 provides the expected cost for the eight pathways in the decision tree in Figure 3.
As mentioned before, expected cost can be calculated by folding back a tree, where each node is replaced by its expected value. Another way of calculating the expected cost, a method that is easily implemented within Excel, is to calculate the joint probability of events within each pathway and multiply this probability by the total costs incurred during the pathway. For example, in Figure 1 there are four pathways, each with a probability of occurring and corresponding program and consequence costs. The expected value of this tree can be calculated by first calculating the expected cost of consequences by multiplying the cost of each consequence by its probability and summing across all consequences. Next, the expected cost for each pathway is calculated by multiplying the probability of the path by its total costs (sum of program costs and expected cost of consequences). The expected cost of each alternative is calculated by summing the expected cost of each pathway that emerges from the alternative.
The expected cost for seamless and traditional probation was calculated in three steps. First, we calculated the expected cost of consequences by multiplying the probability of each consequence by its costs. This is shown in columns 4 through 10 in Table 10. Next for each pathway, the expected cost of the pathway was calculated. This was done by summing the cost values reported in rows in Table 10 and multiplying the total by the joint probability of events in the pathway (product of probabilities in column one and two). Finally, the expected cost of seamless probation was calculated by summing the cost of pathways that followed from joining seamless probation (rows five through 8 in Table 10). The expected cost of traditional probation was calculated by summing the cost associated with the pathways that followed traditional probation (rows 1 through 4). The expected cost for seamless probation was $38.84 per follow-up day per client. The expected cost for traditional probation was $21.60 per follow-up day per client. The net difference was $6,293 per client per year. As anticipated, seamless probation led to reduced arrest rates; this led to $2.31 reduction in expected cost per client per follow-up day. This cost savings was not enough to compensate for the increased cost of mental hospitalization ($13.50 per client per follow-up day), increased cost for delivery of seamless probation ($2.58 per client per follow-up day), additional cost due to use of prison/jail ($2.08 per client per follow-up day), and increased cost of providing treatment ($1.24 per client per follow-up day). Therefore, co-locating the clinic within probation had not led to costs that were less than traditional probation.
Numerous assumptions were made concerning the estimation of costs and probabilities. Concern over the accuracy of these estimates leads one to conduct sensitivity analysis. For each parameter in the decision tree, a breakeven point is found: the parameter is changed until the conclusion is reversed. The percent of change to reach the breakeven point is reported. In addition, where alternative estimates were available, the alternative estimates are used to see if conclusions change.
Decision makers are often concerned how much confidence they should put in the analysis. Statisticians answer these concerns by measuring statistical significance of differences between new clinic and standard care. In decision analysis, one way to help decision makers gain confidence in the analysis is to conduct sensitivity analysis. First, single parameters are changed. Then two parameters are changed at same time, finally multiple parameters are changed. If conclusions are insensitive to reasonable changes in the parameters, then decision makers gain confidence in the analysis.
Table 11 shows the sensitivity of the conclusion in the seamless probation case to changes in rates of any one of the consequences. There was no reduction in rate of any single adverse outcome that could make seamless probation more cost effective than traditional probation (e.g. even when arrest rate of the seamless population was set to zero the traditional probation was still more cost effective). The breakeven points were examined for simultaneous changes in several variables. A 54% reduction in all adverse outcome rates would have made seamless probation more cost effective than traditional probation. The analysis was most sensitive to reductions in arrest, mental hospitalization and incarceration rates. 57% reduction in these three rates would have made seamless probation more cost effective. When examining the sensitivity of the conclusion to simultaneous changes in two rates, the analysis was most sensitive to changes in mental hospitalization rates and incarceration rates. A 69% reduction in these two rates would have made seamless probation more cost effective.
The sensitivity of conclusions to changes in any one of the estimated daily costs was examined. Note that both the traditional and seamless probation have the same daily cost for all adverse events and therefore small changes in these estimates are unlikely to affect the difference between the two groups. For example, the daily cost of arrest is the same for both seamless and probation. The cost of arrest had to increase by 8 fold (from $6,818 to $57.721) before the conclusion that traditional probation is more cost effective is reversed. This analysis suggests that small variations in daily cost estimates of adverse outcomes were unlikely to affect the conclusion.
This chapter has show how cost and effectiveness of a clinic can be examined. A decision tree allows the calculation of cost effectives to be broken down into several estimates: assessing daily probability of enrolling in clinic services, daily probability of facing various consequences, daily cost of clinic operations, and daily cost of various consequences. The later, is available through the literature and the former variables can be measured through tracking a large cohort of patients, through asking time to events of interest or through subjective estimates of experts familiar with the clinic operations.
The advantage of decision analytic evaluation of a clinic is that it reduces the number of estimates needed as daily cost of consequences can be obtained from the literature. In addition, sensitivity analysis could be used to understand how conclusions might depend on various estimates. When conclusions are sensitive to the estimated model parameters, then additional data should be collected to improve the precision of the estimates.
We showed the application of the concepts to measurement of cost effectiveness of substance abuse clinic coordinated with probation, the so called seamless probation. The total cost of seamless probation exceeded traditional probation by $6,293 per client per year. Sensitivity analysis suggested that the analysis was not sensitive to small changes in the estimated parameters.
Send your response by email to your instructor. Include both the question and the answers in your response. Include your contact information.
Please note that this assignment is not currently assigned to students in HSCI 730.