Password Protected Document

Tutorial on Risk Adjusted P-charts

Version February 7, 2001.  This paper has been published.  For a complete reference see Alemi F, Oliver D.  Tutorial on risk adjusted p-charts.  Quality Management in Health Care 2001, 9:4.

            Continuous Quality Improvement (CQI), evidence based medicine, and internal politics of large organizations encourage healthcare leaders to discipline their intuition and rely on data when making decisions that may impact on patient care and health services provided.  Healthcare leaders are advised to use statistical tools such as control charts to see if process changes have led to improved patient outcomes.  To help in this endeavor, several books are available that describe how to construct control charts [i].  But surprisingly, none of these books describe how to adjust control charts for severity of the patients’ illness on admission.  This paper will instruct the reader on how to create a risk adjust control chart.  An earlier publication discussed risk adjusted X-bar charts [ii], useful for analyzing patient satisfaction or health status.  Here the focus is on risk adjusted P-charts, useful in the analysis of dichotomous outcomes such as mortality or other adverse health outcomes. 

            This paper is based on an earlier work by Alemi, Rom and Eisenstein [iii].  Other approaches, e.g. time series analysis [iv], are also possible but more difficult to interpret and use in every day practice. 

What is severity of illness?

                Severity of illness measures patient’s prognosis under ideal care settings.  It provides a quantitative estimate of risk of adverse health outcomes for the patient.  In this paper, the words prognosis, risks and severity of illness are used interchangeably to refer to expectations one has about the patient on admission.  There are numerous approaches to measure severity of illness [v]^{, [vi].}   Some rely on diagnoses codes in administrative databases to construct severity of illness measures; others rely on key clinical findings.  Still others rely on clinicians’ consensus ratings of the patients’ chances for recovery.  No matter how severity indices are organized, investigators validate these indices by predicting patient outcomes.  Thus, when validated, severity indices can be used to assign a probability of an adverse outcome to patients on admission.  There is considerable debate regarding which specific severity index is best [vii].  The authors do not recommend the use of any specific approach to measuring severity of illness and assume that the reader has selected or created one such index that makes sense to his/her clinicians and has high predictive accuracy.  Using the index, the reader is assumed to be able to predict the expected probability of adverse outcomes for each patient.   

Need for severity adjustments

Unlike manufacturing, the healthcare industry deals with a variable input: Patients differ in their severity of illness on admission.  This variability on admission affects care outcomes [viii]. Therefore, it is imperative to adjust for patients’ risk for adverse outcomes.  An example can demonstrate this point.  Consider a hospital that has reduced cesarean rates through educating their patients.  At the surface, this looks like an impressive change in practice patterns.  But is it? Not so, if the aggressive educational campaign has changed the hospital’s reputation.  Now patients who do not have complicated pregnancies are attracted to the hospital.  In short, it is not that processes of care have improved but that patients with less complicated pregnancies are attracted to the hospital.  By blindly applying methods of manufacturing to health care, we might mislead ourselves.

Purpose of risk adjusted control chart

Control charts compare the outcomes of a process to its historical patterns.  The reference point is the outcomes of the same process at some earlier time period.  In risk adjusted control charts, the reference point changes.  Instead of comparing the outcomes to historical patterns, outcomes are compared to what could be expected based on patients’ severity of illness on admission.  The purpose of risk adjusted P-chart is to detect time periods in which care outcomes do not correspond to our expectations at admission. 

Elements of a control chart

A control chart plots data over time.  The X-axis shows time, while the Y-axis shows the probability of an adverse event occurring.  Observed rates of adverse events are plotted against time.  Upper and lower control limits are calculated and are superimposed on the chart.  These control limits are calculated so that 95% or 99% of what we expect falls within these limits.  If data points fall outside the control limits, we assume that the process does not fit our expectations and that some action is necessary.  In these circumstances, an investigation is warranted to find the causes for the process not performing as expected.  All other variations could be assumed to be random chance events and not an indication of changes in quality of care.

Case Example

The interdisciplinary safety committee at a nursing home was concerned with approximately 50% increase in the number of falls in the facility [1].  Many potential contractors judge the quality of nursing homes based on the facility’s incidence of falls.  The safety committee recommended to the Executive Director to create a massive organizational wide educational awareness program. The Executive Director, as the decision maker, wanted to know why the facility had experienced a 50% increase in its fall rate.  Had the facility become negligent in its care or was the facility caring for much sicker patients?  To answer these questions more data were needed.  The safety committee began collecting data that it would need to prepare a risk adjusted P-chart for data analysis.  The data included total patient census during the monitoring period, documented incidents of falls, and each patients’ fall risk assessment score.  In most nursing homes, a licensed nurse documents patient’s fall risks in the medical record.  Risk assessments are done on admission and periodically thereafter.  In this facility, the fall risk assessment tool predicted the patient’s likelihood of a fall based on several criteria such as the ability to ambulate, steadiness of gait, presence of certain chronic medical conditions, number of medications, mental status, and history of falls.  Logistic regression can be used to regress patient care outcomes on the fall risk factors.    Another alternative is to rely on consensus among clinicians regarding the patient’s probability of fall.  Figure 2 shows the data we will use to construct the control chart.  Note that for each patient, in each time period, we have calculated the expected probability of fall.

Steps in constructing control charts

            After data collection, the following seven steps are recommended in constructing a risk adjusted control chart:

1. Check assumptions
2. Calculate observed rates and plot them
3. Calculate expected rates and plot them
4. Calculate expected deviation
5. Calculate control limits and plot them
6. Interpret findings
7. Distribute findings

Step one:  Check assumptions 

            The following four key assumptions must be met.  First, the outcome of care must be a mutually exclusive and exhaustive dichotomous variable.  This means that only two outcomes should be possible.  Furthermore, both cannot occur at the same time and one outcome must happen.   For example, mortality might be considered a mutually exclusive and exhaustive outcome:  There are two possible levels, dead or alive, both cannot occur and one is either alive or dead.  Similarly one can think of falls in the same fashion.  There are two outcomes (fallen and not fallen), it is not possible to be both, and either the patient has fallen or has not.

Second, observations must be made over time.  Control charts are used to analyze data over time. Other simpler approaches are available for data that are not collected over time.

Third, there must be sufficient cases in each time period to detect at least a handful of adverse outcomes.  When the probability of an adverse outcome is very small, e.g. less than 0.05, then it is often not practical to use P-charts.  With such a low probability, more than 100 cases will be needed to detect 5 adverse outcomes.  When adverse outcomes are rare, large databases are needed.  

Fourth, patients’ outcomes must be independent of each other.  It is possible that when a patient falls others are more careful.  If this is true then the probability of fall of a patient affects another.  But in general, it may be reasonable that one patient’s fall is unlikely to be known by another.  This is not true for other illness. Assumption of independence is clearly violated when dealing with infectious diseases. 

Step two:  Calculate and plot observed rates

To calculate the observed fall rate, divide the number of falls in each time period (O_i) by the number of cases in the same time period (N_i).  Thus, the observed fall rate (P_i) can be calculated by using the formula: 

P_i = fall rate in time period “i”

P_i = O_i / N_i

Table 2 shows the calculated rates of fall in the 9 time periods:

 Plotting the observed fall rates help us visualize the trends in the data (see Figure ), but it is too early to tell whether changes in the data are significant.  To make this judgment, you have to wait till control limits have been calculated and added to the chart.

Step three:  Calculate and plot expected rates

The expected rate of falls is calculated by averaging the expectations regarding individual patients.  The following formula is used: 

E_i = ∑_{j=1, …,Ni} E_ij / N_i 

Where E_ij = Expected fall rate of case “j” in time period “i”.

The expected fall rate can be calculated by using the expected probability of falls for each patient.  For example, for the first time period the expected rate of falls is calculated as:

E_i=(0.25+0.40+0.70+0.40+0.15+0.20+0.50+0.50+0.30+0.20+0.40+0.30+0.45+0.25+0.25+0.40+0.45+0.35+0.25+0.10)/20

            Expected fall rates are added to the chart made earlier to produce Figure 4.

            Note that plotting both the expected and the observed rates on the same chart has helped us get a better understanding of the trends.  Except for month one, the expected rate is above the observed rate.  But are the differences big enough to be important?  Before arriving at that conclusions about this data, control limits need to be calculated and plotted.  

Step four:  Calculate expected deviation

          To calculate control limits, we need to have an understanding of variation that exists within each time period.  This is done by calculating the expected deviation.  The expected deviation of time period “i” is shown as S_i and is calculated as:

S_i = (∑_{j=1, …,Ni} E_ij(1- E_ij ))^0.5/N_i.

 Here are sample calculations for the first time period.  You can follow these calculations by looking at the Table 4.  First, multiply patients’ expected rate of fall by one minus the expected rate.  This is shown at column A in the Table 4.  Next, add these rates.  This is point B in the Table.  Finally, take the square root of the total, point C, and divide the result by the number of cases, point D in the Table 4.  

If similar calculations are done for all time periods, the resulting data will look like Table 5. 

Step five:  Calculate control limits

Upper and lower control limits are set in such a fashion so that a certain percentage of data, usually 95%, fall within these limits.  The equation for the upper control limit (UCL) is:

UCL_i  = E_i  + t * S_i

The “t” in the above equation represents a constant that determined what percent of data points are likely to fall within the control limits.  This value depends in part on the number of cases observed.  Table 6 shows various t-values for different sample sizes (to see a complete and more detailed list, click here):

Note that as the sample size decreases the t-value increases.  In small samples, we have less precision and therefore must set our limits wider apart to make sure that 95% of the data will fall within them.  In our example, the upper control limit for time period one can be calculated as:

UCL = 0.34 + 2.1 * 0.10 = 0.55

Note that the t-value for this calculation was 2.1 because the sample size for time period one was 20 observations.

The equation for the lower control limit (LCL) is:

LCL_i  = E_i  - t * S_i

Because negative rates are not reasonable, when the value of lower control limit is negative, it is set to zero.  For the first time period, the lower control limit can be calculated as:

LCL = 0.34 - 2.1 * 0.10 = 0.13

            In a similar fashion, the upper and lower control limits for all time periods can be calculated (see Table 7)  

Plotting the control limit on top of the previous chart, helps decipher the differences between observed and expected rates. 

 Step six:  Interpret findings

A control chart should speak for itself.  The control limits tell the highest and lowest rates we can expect based on patient conditions on admission.  When points fall outside control limits, then for these time periods our expectations have not been met.  An analysis of the data provided in the above graph reveals that there are no points above the UCL.  However, there are points below the LCL in periods 7, 8, and 9.  These occurrences are lower than what can be expected by mere chance.  They suggest that observed fall rates in these time periods were lower than what we would have expected based on patients’ conditions on admission.  Time periods 7, 8 and 9 deserve more investigation.  The nursing home needs to find out what they were doing that resulted in the drop in the fall rate despite the increase in the patient’s risk for fall.

Step seven:  Distribution of the report

When employees find out that something they did has worked, they get motivated to do it again.  To make sure that improvements in processes of care are widely adopted, it is important to tell everyone in the organization about it.  The mere reporting may help improve outcomes [ix].  One way to report outcomes is to circulate the control chart to various employees and units.  In this fashion, managers can celebrate the success of their employees to improve patient outcomes.  This can be done in many ways.  Control charts can be published in the organization’s newsletter.  It can be put on the Intranet and employees alerted to look at it through an email.  One easy method to distribute control charts is to create a storyboard – a large poster containing the problem, list of possible solutions, the solution selected and the impact it made.  The storyboard can be displayed in an area commonly visited by employees.  For continuous examination of quality improvement, it is preferred that the control chart be displayed over time.  This means that data are added to the chart as time goes on.  In this fashion, it represents a monthly monitoring of the unit’s improvement.  The chart tells an unfolding story of success.

Conclusions

Healthcare leaders are constantly called upon to walk a fine line between haste to get the job done and caution to do it well.  Decision makers are accosted from all directions.  Pressures from budgetary constraints, credentialing agencies concerned with quality indicators, employees and consumers often lead to hasty decision-making based on intuition or inaccurate interpretation of data.  When data are analyzed without regard for severity of illness of patients, not only face validity is lost, but also results may be misleading.  Risk adjusted control charts enable healthcare leaders to compare observed and expected outcomes.  It helps decision makers see patterns in data that they may miss otherwise.  It fulfills the promise of evidence based process improvement by focusing attention on actual patterns and not claims of improvement. 

References