A Survey of 92 Quality Improvement Projects

Joint Commission Journal on Quality Improvement  

Farrokh Alemi, PhD, is Associate Professor of Management, College of Nursing and Health Sciences, George Mason University, Fairfax, Virginia.
Duncan Neuhauser, PhD, is Professor, Department of Epidemiology, Case Western Reserve University, Cleveland Ohio.
Farhad Safaie, PE, is Chief Executive Officer, Benchmarking Group International, Inc, Reston Virginia.
Please address requests for reprints to Dr Alemi, College of Nursing and Health Sciences, George Mason University, 4400 University Drive, Fairfax, VA 22101; phone 703 993-4226; e-mail Instructor.

Article at a Glance

Background: Studies focusing on the impact of improvement efforts on the organization have yielded mixed results, which have increased interest in comparing the processes of improvement used. Data for a convenience sample of 92 quality improvement (QI) projects in 32 organizations were gathered from interviews and self-reported surveys from 1998 to 2000. A self-administered questionnaire was developed to measure 70 characteristics of improvement projects. 
Results:  Most (80%) of the improvement projects were conducted by hospitals or clinics affiliated with hospitals. Projects took an average of 13 months from the team's first meeting to the end of the pilot study. During this time, teams met 14 times (approximately once a month) and spent 1.5 hours per meeting. Projects differed in steps they took and results they achieved.  The paper presents data on project’s planning efforts, problem-solving efforts, meeting styles, teamwork, data collection, and outcomes.  Projects had more impact on patient satisfaction and quality of care than organization’s market share or cost of delivery.  Discussion: Patients and employees may be benefiting from improvement projects, but organizations may not be leveraging these improvements to reduce cost of delivery or increase market share. Considerable variation in the projects' impact raises the question of the need to improve the improvement methods. Generalization from this study should be made with caution, as data were based on a convenient self-selected sample of organizations. Furthermore, respondents did not complete all items, and missing information may affect the conclusions. The data on current improvement practices that are provided in this study can serve as a baseline data against which rapid improvement efforts can be judged.

Introduction

In recent years, a number of studies have assessed the impact of improvement efforts on the organization.^1-7 The findings have been mixed, some showing that patient outcomes are more likely to be improved when organizations implement process improvement. Others show no difference among organizations that do or do not implement process improvement. Such variations in the results have increased interest in examining the processes of improvement that organizations use. This article, based on 3 years of data collection, treats the project as the unit of analysis to describe a variety of improvement efforts and their impact on the organizations that sponsored them. In contrast to current studies of the impact of process improvement, the focus is on the improvement method rather than the clinical process and patient outcomes, on the steps involved in the planning and execution of the projects rather than the best clinical practices.

Methods

Source of Data: We based our analysis on a convenience sample of 92 improvement projects in 32 organizations. The characteristics of the organizations included in the study are reported in Table 1. Most (80%) of the improvement projects were conducted by hospitals or clinics affiliated with hospitals, and the organizations reported an average of 7 years of using CQI.

Methods of Data Collection

From 1998 to 2000, we asked health administration, medical, and nursing students in our interdisciplinary quality improvement (QI) classes at Cleveland State University (Cleveland), Case Western Reserve University (Cleveland), and George Mason University (Fairfax, Va) to interview improvement teams in various organizations and report the performance of process improvement projects. We also asked participants in daylong industry conferences on rapid improvement techniques in Iowa to describe their own improvement teams.

Survey Questions

We developed a self-administered questionnaire to measure 70 characteristics of improvement projects.   We also developed an accompanying manual.  This questionnaire and the manual are in public domain and can be used without royalty.  Sample questions are provided in Table 2:

We did not conduct a test-retest reliability study of the questionnaire. However, we did modify it after piloting it with four projects before starting the data collection to reduce differences in interpretation. 

Results

Time Spent on Improvement  

Some organizations abandon QI efforts out of frustration because it takes so long to get anything meaningful out of it.⁸ We collected data on the time it took for project teams to complete their tasks. Across 41 projects on which we had start and end dates, it took 504 days (approximately 17 months; range, 42 days-10.80 years; standard deviation [SD], 828 days) from the identification of the problem to the completion of the first pilot improvement—the so-called first tangible result. Because some projects had not finished, this estimate may change when all projects report their end date. When asked if the pace of improvement was slower than expected, most said no, which leads us to conclude that many may have accepted the 17-month period as the norm for improvement. Responses for 67 projects indicated that of these 17 months, 104 days (3 months; range, 0–2.6 years; SD, 209 days) were spent thinking through and organizing the effort and inviting the improvement teams. A key defining point for projects is the end of the first pilot, when either data on progress have been collected or a second cycle of improvement has started. Respondents took an average of 392 days (13 months; n = 46; 31 days–10.8 years, SD, 779) to progress from the first meeting of the team to the end of the first pilot. What do project teams do during these months? Figure 1 (p 000) shows that among the 92 projects we examined, teamwork and the use of flowcharts and storyboards were prevalent in most projects but that only a minority of the 92 projects repeated cycles of improvement, rolled out their change to the rest of the organization, or celebrated the success of their improvement efforts. 

Patterns of Problem Solving

There is little literature on what works in defining problems, but the few studies that exist suggest the steps one can take to improve problem statements.⁹ A good starting point is to state problems in terms of the patients' experience,¹⁰ which avoids two deadly sins: blaming employees and embedding a solution inside the statement of the problem. Among our projects, 64% (n = 89) of the problems were externally focused (that is, focused on customers' experiences), as opposed to being internally focused on employees’ issues. One way to improve problem statements is to make sure that they describe the problem and not a potential or a favored solution. Among our projects, 22% of the statements represented genuine searches for a solution, as opposed to tools of co-opting employees into a solution perceived by others. Restating problems as an opportunity could accentuate both the positive and the negative aspects of a problem. The literature, reviewed elsewhere, showed that such restatements provide an expansion of the scope of the problem for team members to examine¹¹; 22% of the restatements represented their problem both as a gap and as an opportunity. Some projects focus on clinical problems without management input. This is unfortunate because it fails to take advantage of the organizational view that managers bring to clinical problems. Interdisciplinary input from both management and clinical perspectives could expand the pool of information available to the project teams. In the projects surveyed, 17% (n = 90) had both management and clinical input. Many QI teams choose to focus on small and doable but not central problems common across the organization. It is conceivable that some central problems could be solved quickly, but for the most part, organizational problems are large in scope and difficult to solve. QI teams sometimes choose easy problems to solve because they wish to have small and early successes to generate a continued effort. According to the self-report of project leaders, 35% of 89 projects focused on issues that were central to the organization’s mission.

Making Meetings More Effective

A nagging problem with QI is meetings and more meetings, which consume  a lot of time. The surveyed teams met an average of 14 times per project (n = 75; SD, 18), and each meeting took 1.5 hours (n = 87; SD, 1.5). On average, 62% of the projects (n = 90) judged their meetings to be short and well organized, and 53% judged them to be more productive than expected. Among the projects, 59% were judged to be more task-oriented than social and fun.

Several recommendations on how to make meetings more effective can be found in the literature:

• Distribute an agenda before the meeting starts¹²;
• Have a person facilitate team meetings¹³;
• Poll members (about their views on key issues) before meetings¹⁴; and
• Postpone the evaluation of ideas.¹⁴

Table 3 shows the extent to which projects followed these recommendations.

Methods Used for Problem Analysis and Planning

There are two distinct methods of planning—focusing on "what is" and how to improve it or focusing on "what could be" and how to reach it. In the latter approach, one generates solutions before understanding the constraints of the process in detail. In this way, one's imagination is not restricted with "what is" and can be more expansive.¹⁵ "When people work backwards from what is really desired, they develop energy, enthusiasm, optimism and high commitment."^{16(p 283)} In 9% of the 92 projects, team members arrived at solutions before processes were charted. This suggests that they started with thinking through what the current situation is before they focused on what the future could be. In only 52% of 90 projects did the teams describe the current situation in detailed flowcharts, which took an average of 75 days (n = 65; SD, = 164).

Data Collection Methods

     Of the 88 projects that reported, 79% collected data to examine whether the change they had introduced was an improvement. The process of data collection took 62 days (n = 48; SD, 92). Sampling can reduce data collection time. By focusing on a representative sample, fewer patients are contacted, less time is wasted, and fewer data are analyzed. Among the 66 cases in which data were collected, sampling was done in 17% of the projects. One way to speed up data collection is to rely on numerical estimates offered by persons close to the process. Given time and resource pressures, staff observations of patients may be a reliable source of data that could replace the seemingly more "objective" surveys of patients.¹⁷ Of the 66 cases in which data were collected, 8% relied on subjective estimates to reduce data collection. Still another way to speed up data collection is to plan for the effort before actual data needs are known. For example, teams could put employees on notice that they are about to receive a questionnaire from the team and that when they receive it they should conduct a brief survey and report the results to the team within hours of the request. Of the 66 projects in which data were collected, 17% planned for data collection ahead of their needs. 

Rollout Methods

     Once an improvement has been made, the organization can attempt to make the transition from small-scale to system wide implementation, as did 26% of 92 projects.  This rollout effort took 45 days (n = 19; SD, 52 days). One way to expedite rollout is to use cross-functional teams that include a broad organizational membership, as did 52% of 83 projects. Sometimes the membership of teams is changed so that more people can participate. In the remainder of the projects, teams were either clinical in background (45%) or nonclinical (4%). Another way to expedite rollout of projects to the rest of the organization is to use a storyboard to communicate the improvement effort's impact. Forty-eight percent of projects either did not use a storyboard or did not display the storyboard until the improvement task was completed. Storyboards that unfold over time may engage employees' imagination early, before a solution is reached. The more employees who are involved, the more likely it is that they would implement the team's suggestions.¹⁸ Still another way to motivate other groups to adopt improvements in one unit of the organization is to go beyond rational arguments for change. Most QI teams believe that if they suggest improvements that are in the interest of the organization and in the self-interest of the employees, then these improvements will be carried out. The surveyed teams tried to use several strategies for promoting change, which we categorized as self-interest arguments. Among the 91 reporting projects, 22% of the time the teams tried to persuade others to change by providing them with written reports of the project; 55% of the time they walked key employees through the report. Sixty-one percent of the teams reported changing work norms, for example changing discharge procedures, to encourage adoption of the change. Spreading information about improvement outcomes is one way of encouraging change, but other methods reported by 91 projects include using the organization’s communication channels (emails, newsletters, etc.) 35%) or social gatherings (12%) to provide support for change, making symbolic changes, leading by example at top levels of the organization (16%), reminding key decision makers of the importance of change (37%), and adjusting departmental budgets 14%). One sure way to encourage change is to get early adopters to speak to others about it (35%).

Impact on Performance

Questionnaire items asked survey participants to report the impact of their improvement efforts on cost of care, patient satisfaction, access to care, market share, mortality, morbidity, and employee work life. Since some projects we examined did not measure outcomes, it is important not to mistake failure to report with failure to have an impact. To clarify this issue, in discussing the outcomes of projects we delineate the percentage of projects that did not measure the outcome.

Cost. Across the 92 projects, a small percentage of improvement efforts resulted in tangible cost savings. Only 33 (36%) were intending to save costs, 27% of which collected no data on cost savings, 33% reported it was too early to see the cost savings, 27% reported they have saved potential future costs, and only 6%, or two projects, reported that they have reduced current costs as reflected in their budgets (Table 4, p 000). In summary, most projects did not intend to save costs and among those that did, most did not succeed.

• Client satisfaction. Among the 92 improvement initiatives, 70 (76%) were targeted to improve clients’satisfaction with services, of which only 11 projects (16%) were successful — with an average improvement of 32% (SD, 28%).
• Sales or market share. Nineteen (21%) of the 92 projects had a focus on improving sales or market share, of which 7 projects (37%) reported an average of 28% (SD, 27%) improvement in market share.
• Patient services. The majority—56 (61%)—of the 92 projects reported improvements in patients’ outcomes, including access and morbidity and reductions in patient anxiety, waiting time, and use of patient restraints.
• Employee work life. Among the 92 projects examined, 46(50%) improved employees' work life, of which 48% made the work more convenient, 43% made work less redundant, 59% better defined employee roles,and 59% made employees more aware of each other’s work.

Discussion

In one of the few large comparative studies of improvement that use the projects as unit of analysis, we have provided a method for comparing improvement projects across organizations. The inclusion of the data collection tool (Appendix) can help others conduct similar studies. The data suggest that patients and employees may be benefiting from improvement projects but that organizations may not be leveraging these improvements to reduce cost of delivery or increase market share. Similarly, when chief executive officers and directors of quality assurance/improvement departments in 61 hospitals were asked about the impact of their own QI efforts, they perceived their efforts leading to better patient outcomes but not to better financial outcomes.¹ QI experts complain of the lack of sizable impact of process improvement efforts;¹⁹ the data suggest that in some areas they are right. In addition, we found considerable variation among projects in terms of their impact on outcomes; randomized trials of continuous quality improvement have also shown mixed results.^1,2,20 Such extensive variation in the projects' impact suggests a need to improve the methods of improvement. Caution is warranted in generalizing from this study, since the data were based on a self-selected, convenience sample of organizations. In addition, the projects chosen may not have been representative of the rest of the organization. More than half of the projects reported were still underway, resulting in improved recall of current practices but incomplete data regarding long-term impact. For these projects, we asked respondents if it was too early to report end results. Despite this correction, it is possible that if longer-term data had been available the findings would have been different. Longer-running projects are more likely to be underway at the time of study and therefore possibly more likely to have been included—and more likely than shorter projects to have a large impact. The self-report nature of the data may reflect biased recall of successes and failures. Furthermore, respondents did not complete all items, and missing information may affect the conclusions.

What Makes for a Successful QI Project?

Although 92 improvement projects are not enough to reach conclusions about how successful and less successful ones compare, the findings do suggest hypotheses that can be tested in future studies. The data suggest a number of problems with how improvement teams select an area to work on. Many organizations do not focus on centrally important issues. People inside the organization may rate projects as less serious than do people outside the organization.²¹ It is possible that these employees are working on important problems but do not see it that way. If the perception of these employees is valid, then we need to refocus improvement efforts on central problems, even if such problems are difficult and intractable. If improvement teams can focus better, we may expect better results. The length of the projects varied widely, as Sales et al also found in a survey of projects at 31 hospitals, where length ranged from 1 month to 66 months—and costs ranged from $148 to $18,590.²² Wide variation in project length points to opportunities for improving project management. Data on length of projects can be used in identifying the factors that contribute to longer efforts and in design faster cycles. For example, our data showed that  there was roughly a month delay between meetings. Why is the time between meetings so long and could we meet more frequently? Unfortunately, we do not have the data for why these delays occurred, but perhaps many tasks needed to be done between meetings or the improvement efforts were not considered high priority (because they were not dealing with organizations' central issues). The other way to reduce time spent on improvement is to make meetings more efficient. Could the teams have met for less than an average of 1.5 hours for each meeting? The data showed that some of the teams took a number of positive steps to make meetings shorter and more effective. They distributed agendas before the meetings. Some even polled team members about their views prior to the meeting. But teams did not employ one of the simplest and easiest methods of making meetings shorter and more productive— postpone evaluation until all ideas are expressed.¹¹ The findings of this study can be compared with the experience of the Institute of Health Improvement (Boston) Breakthrough series, in which teams from multiple organizations meet regularly to define and address common problems. Data show that Breakthrough projects take less time per improvement (15 months for 18 improvements),²³ achieving more rapid cycles through a number of steps, including reduced data collection. It is important to gather additional information on the difference between Breakthrough projects and other improvement efforts.

Most of the teams represented in the survey reported here used  flowcharts, which prolonged the improvement efforts by an average of 2½ months. We do not have sufficient data to examine the advantage of conducting flowcharts. As we collect additional data we should be able to address whether teams that spend more time on flowcharts had better outcomes. It is possible that postponing flowcharts until a solution is selected, as entailed by Nadler’s IDEAL system design,¹⁵ may make improvement efforts faster and more radical. Most of the teams also collected data, which on average took more than 2 months. As mentioned earlier, Breakthrough projects do not collect data before generating solutions. Avoiding data collection or merging data collection into other activities already underway may reduce length of improvements. If data have to be collected, effort should be made to make such data collection efficient. Most of the teams did not follow techniques for reducing data collection effort. For examples, teams did not use sampling and did not plan for data collection beforehand. Nor did they rely on subjective estimates from process owners, an approach that could decrease the data collection effort. Surprisingly, most of the teams did not even try to roll out their improvements to other organizational units, perhaps because they did not consider their problem relevant to other units or because it was too early to do so. Improvement teams that did try to roll out their success to other units did so using very traditional methods. They motivated others based on appeals to rational arguments and by making policy changes. Other approaches (for example, use of communication channels, social networks, learning by imitating) were not used as frequently. Given how difficult it is to persuade others to adopt new changes, it is important to use a wide variety of methods for motivating change.

Future Directions 

The study documents a large amount of variation in improvement methods and success rates. It is natural to ask which practices lead to success. One could divide the sample into successful and failed projects and examine which steps are more likely to lead to success. The current sample size is too small to do so. Therefore, this article describes the process of improvement but does not address what leads to success. We will continue our data collection and when we have sufficient data, we will report on factors associated with success of improvement projects. We should be able to determine whether specific approaches to improvement do in fact result in have shorter cycle times and better organization wide outcomes. Much advice has been offered on how to conduct QI faster and better,^{11, 24-27} but ^,[i],[ii], critics are asking for the evidence that proposed methods of speeding up improvements work.²⁸ The data  on current improvement practices that are provided in this study can serve as a baseline data against which rapid improvement efforts can be judged. As our database grows we will be able to report which of the various practices are having an impact in the field.  To facilitate this process we have organized a web site where organizations can report on their improvement efforts.  Our hope is to guide process improvement based on what works.

References

1. Shortell SM, et al: Assessing the impact of continuous quality improvement/total quality management: Concept versus implementation. Health Serv Res 30:377–401, 1995.

2. Curley C, McEachern JE, Speroff T: A firm trial of interdisciplinary rounds on the inpatient medical wards: An intervention designed using continuous quality improvement. Med Care 36(8 Suppl):AS4–AS12, 1998.

3. Shortell SM, et al: Assessing the evidence on CQI: Is the glass half empty or half full? Hosp Health Serv Adm 40(1):4–24, 1995.

4. Gerowitz MB: Do TQM interventions change management culture? Findings and implications. Qual Manag Health Care 6(3):1–11, 1998.

5. Goldberg HI, et al: A randomized controlled trial of CQI teams and academic detailing: Can they alter compliance with guidelines? Jt Comm J Qual Improv 24:130–142, 1998.

6. Shortell SM, Bennett CL, Byck GR: Assessing the impact of continuous quality improvement on clinical practice: What it will take to accelerate progress. Milbank Q 76:593–624, 1998.

7. Gann MJ, Restuccia JD: Total quality management: A view of current and potential research. Med Care Rev 51:467–500, 1994.

8. Senge P: Quality management: Current state of the practice. Keynote speech at the American Quality Congress Boston, October 1993.

9. Smith GF: Defining real world problems: A conceptual language. IEEE Transactions on Systems Man and Cybernetics 23:1220–1234, 1993.

10. Ellwood PM: Outcomes management: A technology of patient experience. N Engl J Med 318:1549-1556, 1998.

11. Alemi F, et al: Rapid improvement teams. Jt Comm J Qual Improv 24:119–129, 1998.

12. Rotondi AJ: Assessing a team's problem solving ability: Evaluation of the Team Problem Solving Assessment Tool (TPSAT). Health Care Manag Sci 2:205-214, 1999.

13. Higgins SE, Routhieaux RL: A multiple-level analysis of hospital team effectiveness. Health Care Superv 17(4):1-14, 1999.

14. French WL, Bell CH: Organization Development. Englewood Cliffs, NJ: Prentice Hall, 1995.

15. Nadler G: Work System Design. New York: Irwin, 1970.  

16. Weisbord MR: Productive Workplaces. San Francisco: Jossey-Bass, 1987.

17. Alemi F: Subjective and objective methods of program

evaluation. Evaluation Review 11:765–774, 1998.

18. McGrath, JE: Interaction and Performance. Englewood Cliffs, New Jersey: Prentice Hall, 1984.

19. Blumenthal D, Kilo CM: A report card on continuous quality improvement. Milbank Q 76:625-648, 1998.

20. Solberg LI, et al: Failure of a continuous quality improvement intervention to increase the delivery of preventive services: A randomized trial. Eff Clin Pract 3:105-115, 2000.

21. Sales A, Moscovice I, Lurie N: Measuring seriousness of hospital quality of care issues. Jt Comm J Qual Improv 22:811-816, 1996.

22. Sales A, Moscovice I, Lurie N: Implementing CQI projects in hospitals. Jt Comm J Qual Improv 26:476-487, 2000.

23. Leape LL, et al: Reducing adverse drug events: lessons from a breakthrough series collaborative. Jt Comm J Qual Improv 26:321-331, 2000.

24. Panzer RJ, et al: Fast track: Cost reduction and improvement. Qual Manag Health Care 6:75–83, 1997.

25. Maxwell CI: Beyond quality improvement teams: Sociotechnical systems theory and self-directed work teams. Qual Manag Health Care 1(2):59–67, 1993.

26. Mosel DM, Shamp J: Enhancing quality improvement team effectiveness. Qual Manag Health Care 1(2):47–57, 1993.

27. Carboneau CE: Achieving faster quality improvement through the 24-hour team. J Healthc Qual 21(4):4–10, 1996.

28. Micklethwait J, Wooldridge A: Making Sense of Management Gurus. New York: Random House, 1996.