Statistical Process Improvement
Georgetown University
 

Comparison of Rates


Let us take on the real challenges in healthcare. The life and death issues.  The persistent errors and the big frustrations. Have you ever had a loved one go to the hospital for a minor problem and die? Ever wondered why healthcare does not work smoothly, why wrong side surgeries occur, or why medication errors occur; this week you get to see the data for yourself. This week we learn how to examine changes in mortality, infection and other rates of other adverse events across time, and we continue to download real data from hospital compare site. P-chart and risk-adjusted p-chart are used to analyze the data.  We will look at the assumptions of these charts, and gain hands-on experience in analyzing real data. After this week, you should be able to go to any hospital and examine if they are getting fewer sentinel adverse events.  Well you do not even need to go to the hospital, you can take their data from the Hospital Compare site and analyze it.

Assigned Reading

  1. Chapter 7 in Big Data in Health Care: Statistical Analysis of Electronic Health Record Read►

Presentations

  1. Lecture on P-chart Slides► YouTube► Video► Transcript►
  2. Lecture on risk adjusted P-charts  Slides► YouTube► Excel► Video► Transcript►
  3. Which chart is right? Slides► YouTube► Transcript►

Examples

  • See a recent tutorial on how to create a risk adjust control chart.  More►
  • See published examples PubMed►
  • Michael Cleary, Ph.D. and Six Sigma give a case study for on-time medication delivery. More►
  • See an example of use of control charts in preventing hospital falls.  More►
  • For an example of risk-adjusted quality control charts see Gustafson's approach to infection control.  More►

Assignments

Instruction for Submission of Assignments: Assignments should be submitted directly on Blackboard.  In rare situations assignments can be sent directly by email to the instructor. Submission should follow these rules:

  1. Make sure that any control charts follow the visual rules below:  (1) Control limits must be in red and without markers, (2) Observed lines must have markers, (3) X and Y axis must be labeled, and (4) Charts must be linked to the data. 
  2. The first sheet in the file should be a summary page or submit a separate summary page.  In the summary page you should list how your answers to the question differs from answers provided within the assignment (inside Teach One or other answers).  You must indicate for each question if your control chart is exactly the same as seen in Teach One or other formats.  For each question, you must indicate if the answers you have provided is the same as the answers supplied on the web.  If there are no answers provided, you must indicate that there were no answers available on the web to compare your answers to. 

Question 1: This exercise allows you to examine how quality of hospital care is evaluated through data obtained from electronic health records.  This exercise is discussed in several health administration courses, including courses on quality improvement, informatics, or statistics. In practice, the need to analyze these types of data arises in improvement teams, where managers, clinicians and sometimes patients work together to systematically improve healthcare processes. In this assignment, you are asked to examine if prophylactic antibiotics are discontinued within 24 hours after end of surgery.  Instead of providing you with the data, we ask that you download the data on the web so that you learn more about the source of the data and peculiar organization of the data.  Hospital Compare provides information on many quality indicators.  In this assignment, you are asked to focus on failure to stop use of antibiotics post surgery. The Hospital Compare web site provides the following rationale for why this measure of quality is helpful:  "A goal of prophylaxis with antibiotics is to provide benefit to the patient with as little risk as possible. It is important to maintain therapeutic serum and tissue levels throughout the operation. Intra-operative re-dosing may be needed for long operations. However, administration of antibiotics for more than a few hours after the incision is closed offers no additional benefit to the surgical patient. Prolonged administration does increase the risk of clostridium difficile infection and the development of antimicrobial resistant pathogens."  To complete this assignment complete the following tasks:

  1. Download Hospital Compare data for 2015 and 2016 years.  Merge the file " Timely and Effective Care - Hospital" across all databases for files from the 2 years.  
  2. Create a query for each year.  Select measure ID "SCIP_INF_3"; this measure refers to prophylactic antibiotic use. Select to work with data from "Southeast Alabama Medical Center"; this is provider ID "010001". 
  3. Calculate rate of overuse of antibiotics.  In these data, the field "Score" refers to the number of patients with overuse of antibiotics.  The field "Sample" refers to all selected surgical patients with no evidence of prior infection. The ratio of Score to Sample provides the rate of overuse of antibiotics.  

Construct a control chart showing rate of overuse of prophylactic antibiotic over time in Southeast Alabama Medical Center.  Download the needed data using the following databases (if new data is available, please use them): 

Question 2:  Nursing home and hospital administrators implement many initiatives to reduce falls.  This problem shows how these initiatives can be measured and evaluated.  Prepare a risk-adjusted control chart for data on falls in a nursing home. 

Question 3: Following data were obtained on post surgical infection rates.  Are we having more infections than expected from the patients' conditions?

Week Risk of Infection for each Patient Number Infected
1 0.9 0.8 0.7 0.8 0.9 0.85 6
2 0.7 0.8 0.7 0.6 0.8 0.9 5
3 0.8 0.95 0.92 0.87     4
4 0.5 0.6 0.66 0.67     3
5 0.3 0.4 0.5 0.4 0.5 0.34 2
6 0.3 0.4 0.5       1

Question 4: Download Hospital Compare data for the year 2015 and 2016.  Start from 1/1/2015 till the most recent available database. Since the data on AMI payments does not change in the files, you can save time and read only the following 3 files into a new database:

File Database Field Start End
HQI_HOSP_AMI_Payment   HOSArchive_20150122 Compared to National 2010 2013
HQI_HOSP_Payment HOSArchive_20150716 Category 2011 2014
HQI_HOSP_PaymentAndValueOfCare Hospital_20161110 or HOSArchive_20161110 Payment Category 2012 2015
Different names Different names Payment Category 2016 Now

In these files the denominator indicates the number of patients.  Whether the hospital had above average payments are reported in different fields: [Compared to National], [Category], and [Payment Category]  Select data that meet the following conditions:

  1. Select the "PAYM_30_AMI" measure
  2. Select hospitals that in the denominator do not have "Not Available".

Construct a control chart showing probability of change over time.  Download using the following files:  2016-11-10.zip► 2016-08-10.zip► 2016-05-04.zip► 2015-12-10.zip► 2015-10-08.zip► 2015-07-16.zip► 2015-05-06.zip► 2015-04-16.zip► 2015-01-22.zip►

Question 5: On average 1 in 10 patients have some sort of medication error in our facility; what is the probability that out of 10 patients none will have medication errors? See page 176 of the required textbook for how a similar question was answered.

Question 6: The probability of a patient recovering from a heart operation is 0.9. Assuming that heart operations are not related to each other, what is the probability that exactly 4 out of 8 next patients will survive the operation?  See page 176 of the required book for a similar example. 

Question 7: An insurance salesperson is about to sell life insurance policies to 7 unrelated women, all of whom are of the same age, belong to the same race, and are in good health. According to independent actuarial estimates, the probability of a woman of this age, race, and health status being alive 20 years from now is 80%. What is the probability that in 20 years all 3 women are alive?

Question 8: Healthcare organizations are often interested in assessing the impact of screening efforts in generating new referrals.  Patients may return to many different clinics and not just to the original screening clinic.  They may also return to a clinic months later and not immediately.  The attached data shows visits from pregnant mothers who received ultrasound tests.  Data are presented on date of visits and clinics that were visited.  Re-organize the data to count the number of referrals per month to neurology. Note that you will need to use Excel or SQL text processing commands to identify when the mother has returned to the neurology department. Construct a p- chart for the probability of a subsequent neurological visits in mothers who had an ultrasound screening.

Question 9: This exercise allows you to examine how quality of hospital care is evaluated through data obtained from electronic health records.  This exercise is discussed in several health administration courses, including courses on quality improvement, informatics, or statistics. In practice, the need to analyze these types of data arises in improvement teams, where managers, clinicians and sometimes patients work together to systematically improve healthcare processes.  In this assignment, you are asked to examine percentage of healthcare workers given influenza vaccination for preventive care. This is a patient safety measure for timely and effective care. Following Donabedian’s domains of quality (structure, process, outcome), this measure falls under the process domain. “Process refers to the way in which diagnoses, treatments, practices to avoid harm to patients and other care are rendered – whether steps known to be effective in preventing infections and medical errors, for example, are built into hospital routine.” The CDC states that during 2019-2020 flu season, flu vaccination coverage among health care personnel was 80.6%, similar to coverage during the past five seasons (77.3% -81.1%). The IMM-3-FAC-ADHPCT measure is defined as percentage of healthcare workers given influenza vaccination.

Read the rationale behind this measure. Download the following datasets from the Hospital Compare and note that the measure ID has changed in different years. 

  • For 2015, download HOSArchive_Revised_Flatfiles_20151210.zip and use measure ID: IMM_3_OP_27_FAC_ADHPCT 
  • For 2016, download HOSArchive_Revised_Flatfiles_20161219.zip  and use measure ID: IMM_3_OP_27_FAC_ADHPCT 
  • For 2017, download HOSArchive_Revised_Flatfiles_20171024.zip  and use measure ID: IMM_3_OP_27_FAC_ADHPCT 
  • For 2018, download HOSArchive_Revised_Flatfiles_20181031.zip  and use measure ID: IMM_3_OP_27_FAC_ADHPCT 
  • For 2019, download HOSArchive_Revised_Flatfiles_20191030.zip  and use measure ID: IMM_3 
  • For 2020, download HOSArchive_Revised_Flatfiles_20200422.zip  and use measure ID: IMM_3 

Merge the file " Timely and Effective Care - Hospital" across all data for the 6 files. Select either measure ID "IMM_3_OP_27_FAC_ADHPCT" or measure ID "IMM_3". Analyze the field "Score." In these data, the field "Score" refers to the percentage of workers given flu vaccinations. Construct a control chart showing percentage of health care professional's vaccination over time in "Brookdale Hospital Medical Center" in New York; this is provider ID "330233".  Estimate the control limits based on the "Presbyterian Hospital" in New York; this is provider ID "330101". These two hospitals were selected based on a July 2020 article by the NY Times that described Brookdale Hospital Medical Center as a “struggling independent hospital, located in Brooklyn”.  In the same article, New York – Presbyterian Hospital was described as “Manhattan’s largest private hospital network”. The article claimed that surviving COVID-19 virus might have depended on which hospital admits you

Question 10: Please attach the copy of your second email in the case study.  The overall purpose of the case study is to get you engaged in networking with the industry. You should proceed with your contact, even if they have not answered your first email. You should continue contacts unless you have a bounced email. The 2nd email should focus the analysis by selecting the following:  (1) A type of patient that the analysis focuses on (2) The measures that describe the quality of care for these patients, and (3) competitors selected for the hospital. Case Study►

  1. Choose a type of patient that your analysis will focus on. Choose at least 5 quality measures that are appropriate for a patient with a particular illness. Choose these indicators so that you can create a cohesive picture of the quality for a patient with a particular illness. The quality indicators are listed in the web site on Hospital Compare. Dictionary► List► 
  2. Select the peer organizations (at least 3) for the hospital you have selected.  These are sometimes local competitors and other times national competitors.  Make sure your selection makes sense, in terms of hospital size and market reach. You can look inside your data to see what other organizations are in the same region.  You can look in Linked In (R) to see possible peer organizations.
  3. Email your preceptor and ask his/her opinion about your selection (include this email in response to this question).  Please start with draft email and try to follow it; but add anything that you see appropriate.  Follow the format provided.  Refer to the contact by their qualification, e.g. Dr. Review the email for sentence structure and other issues.  This is how they are going to judge you so the more time you spend on this email the better you will come across to the preceptor. Draft Email► Example from Vivian►

More

  • Risk adjusted probability charts More►
  • Independence of observations  More►

  • This page is part of the course on Statistical Process Control, the lecture on Comparison of Rates.  This presentation was based on Alemi F, Rom W,  Eisenstein E. Risk adjusted control charts for health care assessment.  Annals of Operations Research, 1996. Created on Tuesday, September 17, 1996.  Most recent revision 10/19/22.