Risk Assessment

Introduction

The following lecture is based on Alemi F, Rice J, Hankins W. Predicting in-hospital survival of myocardial infarction: A comparative study of various severity measures. Medical Care 1990; 28 (9): 762-775. Pubmed►

Session Objectives

1. Discuss the role of risk assessment in the process improvement.
2. Describe the ideas behind different approaches to measurement of severity.
3. Contrast the accuracy of severity indices.

Definition of Severity

There are many ways to evaluate the quality of health care services. One increasingly popular way is to examine the rate of adverse health outcomes. Mortality is frequently used to categorize outcomes for patients who have acute, life threatening illness. Within both the group of patients who live and the group that dies, some patient might receive good care while others receive poor care. Some patients may be so ill that they cannot be saved by the best of care. Other patients may get well despite poor care. It is difficult to define what is severity because it is difficult to separate the influence of quality from severity. One definition is that severity is the progression of the disease when left untreated. Since no one in his right mind leaves diseases untreated, it is difficult to observe the progression of the disease. Another definition is that severity is the progression of the disease given customary treatment. This definition is also flawed because customary treatment includes both poor and good quality care. These difficulties of observing severity of illness within one person has led many investigators to suggest that severity is a comparative concept, meaning that a reasonable way to observe severity is to examine people at different stages of the disease. These individuals have different risks for mortality or adverse health outcomes. While it is difficult to observe progression of a disease, it is easy to see that some patients are further along in their illness than others.

Importance of Measuring Prognosis of Patients

In observational data, comparative effectiveness of treatments cannot be understood until patient's prognosis are measured and used to statistically control for observed outcomes.  Then, variations in outcomes that are not explained by the patient's severity are attributed to treatment effectiveness.  When severity of illness is not used, contradictory findings may be discovered.  For example, studies have shown that poor measurement of severity leads to the erroneous conclusion that medications are associated with poor outcomes.  The importance of severity in measuring outcomes has been shown in numerous diseases.   Asthmatics► Hypertensives► Stroke►

Measures of Severity

There are many measures of severity, some commercial and others available through academic journals. Because severity measurements is the reason for attribution of adverse outcomes to poor or good quality, it is important to use a severity measure that is well accepted by the people whom we are evaluating. Hence, many select commercial severity indices. In this lecture we review a sample of these indices and their major differences.

1. Disease Staging index is designed for a broad range of patients and is not limited to specific illness. Disease staging assumes that diseases are first localized and later spread to other parts of the body systems. As a disease advances to higher numbered stages, it is associated with increased risk for the patient. The Computerized Disease Staging has four stages which can be subdivided into additional sub-categories.  Stage 1 includes conditions with no complications and minimal risk for the patient. Stage 2 includes problems which are contained in one organ or system. Stage 3 includes problems in multiple sites and general systemic problems. Stage 4 is death.  As frequently implemented, Computerized Disease Staging assigns severity stages based on diagnoses codes available through billing information. These codes are often based on the International Classification of Diseases, version 9, clinical modification (ICD-9-CM) More►
2. Patient Management Categories is designed for a broad range of patients and is not limited to specific illness. Patient Management Categories specify the costs for different care regimens. A panel of experts were asked to describe the ideal treatment for specific patient groups and a cost was estimated to correspond to the ideal care. Patient Management Categories groups patients based on their diagnoses codes available through billing information. Like Disease Staging it relies on the International Classification of Diseases, version 9, clinical modification (ICD-9-CM). Unlike Disease Staging it produces an interval scale, meaning that the scores assigned preserve the magnitude of the difference in the severity indices. Thus, a score of 10 is not just more ill than a score of 5 but twice as worse off as a score of 5. More►
3. The Acute Physiological and Chronic Health Evaluation (APACHE) index was originally designed for critically ill adult patients but was later proposed for use by patients outside critical care units. The APACHE score is the sum of three components. These are: (a) deviations from norm on 12 physiological variables like heart rate, blood oxygen level, or respiratory rate; (b) age of the patient; and (c) chronic illness, including coma. APACHE is usually measured during the first 24 hours of hospital admission. The most abnormal values during this period are recorded and scored. APACHE produces an interval scale, where the score of 10 is twice as bad as 5.  More►
4. Medisgroup is designed for a broad range of patients and is not limited to specific illnesses. It scores have five levels: 0 through 4.  At level 0, there are no clinical findings. At level 1, there are minimal abnormal findings. At level 2, there are either acute findings or findings with an unclear potential for organ failure. At level 3, there are clinical findings with high potential for imminent organ failure. At level 4, organ failure is indicated.  The Medisgroup scoring does not follow specific mathematical rules like APACHE, where the scores of abnormal findings are added. Instead, Medisgroup relies on artificial intelligence if-then rules to score combination of clinical findings. These types of if then rules create a scoring system that has a lot more flexibility. Medisgroup relies on key clinical findings during the first 24 hours of admission of a person to the hospital. Key clinical findings may be specific laboratory findings or it could be clinical observations. It produces an ordinal severity scale. More►
5. Computerized Severity Index is designed for a broad range of patients and is not limited to specific illnesses. It relies on both ICD-9-CM diagnosis codes and key clinical findings. It produces an ordinal severity index, where for example a patient with a score of 4 is worst than a patient with a score of 2 but not twice as ill. This index scores range from 0 to 4. It begins with the patient's principal diagnosis and uses physiological markers to adjust the diagnosis. The internal working of this index are not publicly available. More►
6. Several indices rely on select categories of diagnoses within administrative databases.  Charlson Index► Elixhauser List►
7. Comorbidity-Polypharmacy Score uses patients medications to measure severity of illness  More►
8. Multi-morbidity index is a comprehensive method of using diagnoses in administrative data to predict prognosis of the patient More►

Best Approach to severity measurement?

Reliance on different sources of information

Severity indices differ in the source of information used to measure severity of illness. Some rely on physiological markers, others on ICD-9-CM codes. Both could be affected by treatment. Physiological markers are affected by treatment on route to the hospital and during the hospital stay. For example, medications may show a normal blood pressure for a patient that minutes before admission was in shock in the ambulance. Similarly, because ICD-9-CM codes are based on the treated diagnosis through out the hospital stay, they may be affected by complications that arise due to poor care. For example, when a patient falls and breaks her hip then this is added to her severity score while clearly the fall was not the condition the patient came in with. Because both sources of data could be affected by treatment, whenever possible both should be used. Diagnoses also reflect a summary of what the clinician taking care of the patient considered most likely reason for admission. Clinical markers, in contrast, are at best a reconstruction of what the documented information implies. Since in busy clinical practice, much of the information is not documented, the use of key clinical findings may not be as good as diagnoses codes.

Differences in scores produced

Severity indices also differ in the type of scores they produce, some produce ordinal scales others produce interval scales. Interval severity scores are most helpful for bench marking, as these numbers can be averaged and used in various control charts. In contrast, ordinal scales must be transferred to interval scales before use in control charts. The following table shows the differences select severity indices:

Accuracy of Predictions

The accuracy of the various severity indices in predicting mortality from Myocardial Infarction in Hospitals in New Orleans area in 1985 was as follows:

These data suggest that the performance of these indices for patients with myocardial infarction may not be substantially better than what can be expected from predicting that all patients will survive. Whenever possible multiple severity indices should be used to improve the accuracy of predictions.   Severity indices to detect problems in quality of care More► 

What Do You Know?

Advanced learners like you, often need different ways of understanding a topic. Reading is just one way of understanding. Another way is through writing. When you write you not only recall what you have written but also may need to make inferences about what you have read. The enclosed assessment is designed to get you to think more about the concepts taught in this session.

1. What problems do you think reliance on ICD-9-cm codes create for measuring severity of illness? Keep in mind that ICD-9 codes are based on the entire course of a patient's illness including possible adverse events due to poor care. 
2. What is an ordinal scale, what is an interval scale, and why would you prefer one to the other?
3. Which do you think is more likely to be affected by treatment, physiological markers collected during the first 24 hour of admission or ICD-9-CM codes classifying the purpose of the hospitalization?  Keep in mind that key clinical findings may be normal if the patient is managed well.
4. Construct a simple Multi-Morbidity Index using the data in the Table below.
  Video► Slides►



Case Type	1st Diagnosis	2nd Diagnosis	3rd Diagnosis	4th Diagnosis	Length of stay	Number of Patients
1	MI	CHF	DM		5.56	10
2	MI			AA	4.10	10
3		CHF	DM	AA	5.54	10
4		CHF	DM		3.56	20
5	MI	CHF		AA	7.03	30
6	MI	CHF			5.02	30
7		CHF		AA	5.04	30
8	MI	CHF	DM	AA	7.62	40
9	MI				2.03	40
10		CHF			3.03	40
11	MI		DM		2.60	50
12			DM	AA	2.57	50
13			DM		0.61	60
14				AA	2.12	70
15					0.01	80
16	MI		DM	AA	4.57	120
MI = Myocardial Infarction; CHF = Congestive Heart Failure; DM=Diabetes Mellitus; AA=Alcohol abuse



1. Assess the average severity of CHF, MI, Diabetes, Hypertension, Alcohol Use, and ACL surgery (assume that sicker patients have longer stays).  
   
   • For example, to calculate the average severity associated with MI, we need to compare all cases of MI to same number of control patients without MI.  Cases and controls should have the same comorbidities.   For example, type 1 patients has the same comorbidity (i.e. CHF and DM) as type 4 patients.  These two types only differ in the presence or absence of MI.  So comparing them allows us to see the contribution of MI to severity. The procedure is to pair all cases with MI against controls without MI that share the same comorbidities.   Then the average impact of MI can be calculated independent of comorbidities.  For example, for MI, we notice the following average length of stay for MI patients with different comorbidities:
     
     

     Comorbidities	Length of Stay
     	Cases with MI	Controls without MI
     CHF,DM,AA	7.62	5.54
     CHF,AA	7.03	5.04
     CHF, DM	5.56	3.56
     CHF	5.02	3.03
     DM,AA	4.57	2.57
     AA	4.10	2.12
     DM	2.60	0.61
     (None)	2.03	0.01
     Average	4.82	2.81

     
     Note each cell above shows the length of stay for average case with MI or average control without MI.  In this format, it is now easy to see that the average impact of MI across the comorbidities is the difference of the red and green columns.  The score associated with MI is 4.82-2.81=2.01. Similar analysis is done for each of the other diagnoses providing us with a score associated with each diagnosis. 
   
   Inside Excel, we do not want to waste time to sort the data into the red and green columns, so we use a function called sumproduct that allows us to take advantage of places where MI is present and where it is absent. 
          
   
   • Assess the overall severity of the 16 types of cases in the Table as the sum of the severity of each of their diagnoses. 
     • So for case type 1 the severity will be the sum of severity associated with MI, plus severity associated with CHF, plus severity associated with DM. 
   • Plot the patient's length of stay against the patient's severity of illness.  This is a scatter plot. 
   • A new patient with MI, AA, and CHF is discharged in 3 days.  Is this length of stay more, or less, than what would we expect for these types of patients?  
   • Can this procedure be repeated in an electronic health record, where there are thousands of diagnoses and millions of cases? 
     

Presentations

To assist you in reviewing the material in this lecture, following presentations are available:

• Lecture on risk assessment Slides► Listen►
• Scoring for Multi-Morbidity Indices  Video► Slides►

Narrated lectures require use of Flash►  

More

• Use of severity of illness to classify and monitor medication errors More►
• Use of pharmacy data to measure severity of illness More  
• Risk of risk assessment More►
• Nursing severity indices More►
• Community wide measures of quality of care More►
• Measures of outpatient severity of illness More►. 
• Severity of episodes of illness More►.