Reading:  Participatory Vendor Selection

• Ask Questions
• Evaluate Lecture
• Course Page
• Syllabus
• Report Errors

Introduction

It is important to engage clinicians, especially front line clinicians, in selection of electronic health records.  People who are likely to use the systems should be part of selecting it, so their needs and preferences are met.  Some investigators have gone as far as recommending that implementation of information systems should be led by clinicians, typically physicians or administrative nurses (Doolan, Bates,  James, 2003). Unfortunately, most clinicians do not have a background in informatics.  Aside from ease of use, they may not pay attention to other factors in selection of information system vendors.  They may be too deeply involved in current practices to notice the potential new ways of reorganizing clinical care. A dilemma arises regarding how to help clinicians participate in vendor selection when they do not have the necessary background.   Even if one wants to actively engage clinicians in the selection of information systems, there are so many clinicians that a logistical problem arises regarding how to solicit the opinions of a large number of clinicians.  This section describes a procedure for informing and encouraging the participation of clinicians in the selection of information systems and their vendors.

When it comes to purchase of information technology everyone in the organization has an opinion.  Clinicians want to be involved because they are going to be saddled with the use of the system; they want to insure it has the features they need.  Managers want to be involved because they want to insure that the new information system solves long lasting quality and productivity problems.  The chiefs (CIO, COO, CFO, etc.) want to be involved because purchase of information systems is a large capital expenditure and they want to insure that money is well spent. With so many different opinions, it is difficult to create a consensus.  This chapter shows you how to achieve a consensus on vendor and product selection.

A New Group Process

There are many processes that can be used to capture a group's decision.  Three and half decades of research on effective teamwork illustrate the following lessons:

• Postpone evaluation. It is best to separate idea generation from idea evaluations. When evaluation is postponed, more ideas and more creative ideas emerge.
• Think again. It is best to think through the decision again, especially when numerical estimates are involved. In repetition, people gain confidence in what they are doing and can see pitfalls previously missed.
• Meet before the meeting. It is useful to get input from group members individually, before they can influence each other.  This can often occur through the use of computers and might be one way of combining computer facilitated decision support with face-to-face meetings.
• Fudge the origin of the idea. It is important to evaluate ideas based on their merit and not based on their popularity.  Successful group processes separate ideas from the originator of the idea.  In this fashion, ideas are judged based on their merits and not who proposed them.
• Instruct the group to behave. It is best to instruct group members to keep calm and accept conflict as productive.  Simple instructions at start of a meeting can set the tone of the group discussion to come.  Successful instructions, proven useful in managing groups, include the following:  avoid arguing, avoid win-lose statements, avoid changing opinions in order to reduce conflict, and avoid conflict-reducing techniques such as majority votes and bargaining. View differences of opinion as natural and initial agreements as suspect.
• Use computers to facilitate components of the meeting.  It is best to use technology to help the groups arrive at consensus but such use should not reduce the rate of exchange of ideas or the ease with which members interact with each other.

Based on the lessons from the literature, Gustafson, Alemi and Cats-Bariel (1992) designed a group process that is commonly known as "The Integrative Group Process (IGP).  This is an eclectic group process.  It borrows from different well-known processes elements that work best:

• Like Nominal Group Process, IGP postpones the evaluation of ideas until the analyst has collected the ideas of all members.  In addition, both approaches improve estimates of relative importance of ideas through repetition (i.e. both require the group member to assess, discuss, and revise their numerical estimates). 
• Like Delphi, IGP obtains remote and private opinions. But unlike Delphi, these remote contributions are followed by face-to-face interactions.
• Like the Group Communication instructions, IGP sets ground rules for free-form group interaction.
• Like Cognitive Mapping and the Social Judgment Analysis, IGP focuses discussion on the group member's reasoning rather than the group's decision.  Thus, group members can better understand why they disagree on a point if they come to see each other's reasoning.
• Like computer facilitated group processes, IGP collects ideas from group members through computers: typically by emails.  IGP and computer facilitated group process differ in what occurs after the initial collection of ideas.  IGP emphasizes face-to-face meeting after computer facilitated collection of ideas. 

Tracking Preferences through Numbers

Everyone has a different set of priorities and preferences.  One way to track the stakeholder's preferences is to come up with a numerical scoring scheme that preserves their preferences.  For example, Multi-Attribute Value models assign higher scores to preferred options.  The purpose of these models is to assign one score that reflects the decision maker's preferences across a large number of criteria.  Information systems differ in many ways.  Making the judgment of which one is the best for the organization requires keeping track of the performance of the system on a large number of criteria.  No one can remember all the nuances and differences.  To simplify the task, Multi-Attribute Value models allow the decision maker to make an evaluation of the system on each criterion and use a mathematical formula to integrate these evaluations into a single score.  If each option was described in terms of n attributes A₁, A₂,  ... , A_n the option would be assigned a score on each attribute, V(A₁), V(A₂),  ... , V(A_n). The overall value of an option equals:

Overall score = Function [ V(A₁), V(A₂),  ... , V(A_n) ]

In other words, the overall value of an option is a function of the value of the option on each attribute.  This equation provides us with a way of scoring the judgments of a single member of the team.  The group's decision could be thought of as the average of the judgments of various members of the team.  If so, each individual scores the vendor based on a set of criteria and the average of these scores is taken to represent the consensus.  Mathematical aggregation of individual's scoring procedures into a group model loses the real advantage of having the group.  We engage a group of decision makers because we want them to arrive at better decisions or at least be vested in the decision and work to make it a success.  Mathematical modeling and scoring undermines both objectives.  To effectively model a group's decision, one should emphasize behavioral methods of arriving at a group consensus.  Mathematical aggregation rules are useful but do not replace the type of consensus that emerges when people talk through their assumptions and differences. 

Vendor selection is much more than scoring the performance of various vendors and their products.  It is an opportunity for discovery and consensus formation.  It gets the organization ready for the upcoming changes.  In the end, the scoring is of secondary value, it is the consensus process that really counts.  If all parties feel that they have been heard and that their concerns were adequately addressed, then they would be much more committed to the decision made.  If consensus emerges, the scores can be abandoned.  It is the formal and informal conversations that help the organization make the right choices, the scores is just a tool towards to goal of helping the organization come to consensus.

In the following, we describe a vendor selection process.  We are assuming that the decision to purchase the information system has been made and the question that remains focuses on which system should be selected from which commercial vendor and with what features.  The process we are recommending to you uses subjective judgments regarding how each vendor performs on various criteria.  The decision to select a vendor cannot be based entirely on objective data.  At some point, because the objective data is missing, or the attribute being measured is inherently subjective (e.g. reputation of the vendor), it is important to rely on opinions of experts. A process that we further describe below.  In addition, the approach we are recommending assumes that quantification and modeling will help. Modeling preferences around vendor selection helps communicate to others why one vendor was preferred to another.  The quantification procedures that we describe allow one to accept losses in one area in lieu of gains in another.  It allows the complex task of vendor selection to be based on a number of smaller decisions about meeting specific criteria.  Clear communication around major investments helps show why a particular system was selected.  Some organizations, however, may prefer leaving their decisions ambiguous.  The process described here is not suitable for situations where clarity is not desirable, or where the decision on how to proceed is obvious (i.e. there is only one vendor for the information system).

Vendors are typically asked for five categories of information:  product functions, corporate information, service, technology and price.  Each of these categories includes a number of attributes.  Cost might be broken into initial licensing fees, maintenance cost, and networking costs.  Product function might have numerous other sub-categories (i.e. scheduling, billing, etc.), each leading to a different set of attributes. The California Healthcare Foundation (Forrester Research 2003) and the Leapfrog group (Kilbridge, Welebob, Classen, 2006) each have developed detailed procedures for vendor evaluations.  These efforts are prescriptive, in the sense that they advise how practices should evaluate electronic health records.  In addition, others have taken a more descriptive route.  They have examined how real practices make vendor decisions (Eden 2002).  Throughout this section, we use the attributes identified in Eden's survey of physician practices to construct an example of types of criteria that might be used in vendor selection. 

Step 1: Involve All Stakeholders

In organizations, there are often many decision makers. No single viewpoint is sufficient, and the analyst needs a multidisciplinary consensus instead.   Obviously, the Chief Information Officer, the Chief Medical Information Officer and the Nursing administrator should be involved.  Clinicians and managers in key information technology (e.g. Computerized Physician Order Entry committee, master patient index committee,  Health Insurance Portability and Accountability Act Committee, etc.) and non-information technology committees (e.g. pharmacy committee, patient safety committee, executive or medical leadership committee, quality assurance committee, etc.) should be involved.  Obviously, people with direct budget authority over the purchase of the new system and those involved in maintaining legacy systems should also be involved.  If the same system is being implemented at multiple site, individuals across sites should be involved.  All of this leads to a very large number of stakeholders.   It is important to engage them all.

It may be useful to remind the invited members about who else is being invited. People are more likely to come to meetings when they see that people they admire are present.  Many clinicians are not paid for administrative meetings; its is important to emphasize how their time will be used effectively.  It also helps to reassure each participant that their views matter and that the decision of how to proceed has not been made:  the process is not a sham and their participation matters. 

Step 2: One-on-One Interviews

Before face-to-face meetings, group members are individually interviewed and their scoring system is modeled.  If group members live far apart the interviews are done by phone, a series of emails, or through computer connections. Whether done remotely or face-to-face, the interview is scheduled ahead of time.  During the interview, which takes roughly one hour, the analyst explains the group's task, elicits the participant's criteria for evaluating vendors and asks the participant to score the relative importance of different criteria. The bulk of the interview time is spent listening to the group member and trying to model the reasoning behind his or her choices.  The process of interviewing a single group member can be broken down into following tasks:

1. Introduce yourself and your purpose. Briefly say who you are, why you are talking to the decision maker, the purpose of vendor selection, and how you will go about using the results of the interview. Be as brief as possible. An interview is going well if the analyst is listening and the decision maker is talking. If it takes five minutes just to describe the purpose of vendor selection, then something is amiss.  The analyst, who organizes the vendor selection process, should be assertive in setting the interview's pace and agenda.  Because the analyst is likely to receive comments after a pause in the conversation, it is important to be judicious about how one ends a comment.  Every comment made by the analyst should follow with a question that directs the decision maker's attention to the next task at hand.  Thus, if the analyst stops after saying, "Our purpose is to select vendors," then the decision maker will likely discuss the purpose.  However, if the analyst immediately follows the previous sentence with a question about the decision maker's experience in vendor selection or information systems, the decision maker is more likely to begin describing his or her background. The point is the analyst sets the agenda and should construct questions to lead discussion through topics needed for the analysis. 

2. Ask the decision maker to introduce himself/herself. It is important to establish that the analyst is a record keeper, a facilitator; the decision maker provides the content and expertise. A good way of doing this is to ask the decision maker to introduce him or herself by describing relevant experiences. This approach clarifies the roles of each party. Whether the analyst explicitly pauses to ask for the introduction or not, most people go through lengths to mention their accomplishments anyway.  Asking for an introduction allows the decision maker to do so at start of the conversation and not throughout the interview.

3. Start with tangible examples. Concrete examples help the decision maker understand which attributes should be used to select vendors and how they can be measured.  The analyst should ask the decision maker to recall an actual situation and contrast it with other occasions to discern the key discriminators. For example, the analyst might ask the decision maker to describe a system that failed and one that served his or her needs.  By contrasting these two systems one can find what aspects were different.  Continue by asking the decision maker to think about specific attributes that can be used to distinguish them until all relevant attributes are identified. Here is a sample dialogue:

In this dialogue, the analyst started with tangible examples and used the terminology and words introduced by the decision maker to become more concrete.  This dialogue is designed to reinforce to the decision maker that the analyst is listening to him.  Therefore, as far as possible, it is important to use the words and terminology used by the decision maker. 

4. Arrange the attributes in a hierarchy. Some analysts suggest using a hierarchy to solicit and structure the attributes (Keeney and Raiffa 1976). For example, the decision maker suggested that costs are important but that there are different types of cost attributes:  cost of purchase, cost of maintenance, and cost of impact on practice efficiency. These three costs can be sub-attributes under the cost category.  The hierarchical structure promotes completeness and simplifies tracking many attributes.   Make sure that all attributes included are relevant to the vendor selection task and that each attribute is different from the other.

5. Assess possible levels of each attribute. The analyst starts by deciding if the attributes are discrete or continuous. Attributes such as cost are continuous; attributes such as compatibility with existing systems are discrete. Continuous attributes may be expressed in terms of a few discrete levels, so that cost can be described in ranges and not actual cost values. There are three steps in identifying the levels of each attribute: 

   1. Define the range of the attribute by selecting the best and worst levels in the attribute

   2. Define some intermediate levels

   3. Fill in the other possible levels so that the listing of levels is exhaustive and capable of covering all possible vendors.

   To define the range, the analyst must examine the best and the worst vendor on the single attribute. Thus, for the cost attribute, the range is set so that it will cover the lowest and highest cost information system.   A typical error in obtaining the best and the worst levels is failing to describe these levels in detail. For example, in assessing the value of the attribute "ease of use," it is not helpful to define the levels as:

   • Easiest to use system

   • Hardest to use system

   The adjectives easy to use and hard to use should be defined.  It is best to avoid using adjectives in describing levels, as decision makers perceive words like easy, or best in different ways. The levels must be defined in terms of the underlying physical process measured in each attribute, and the descriptions must be connected to the nature of the attribute. Thus, a good level for the easy to use attribute might be "Entering the average patient encounter takes less than 2 minutes" and the worst might be "Entering the average patient encounter takes more than 5 minutes."

   Next, ask the decision maker to define intermediate levels. These levels are often defined by asking for a level between the best and worst levels. In the example, this dialogue might occur:

   Analyst:	I understand that in some systems it may take less than 2 minutes or more than 5 minutes to enter information on an average patient encounter.  What about other systems.  Where do you think they would fall in the range between less than 2 minutes to more than 5 minutes.
   Decision Maker:	Well, a host of things can happen. But on average I think in many systems it may take about 3 minutes to enter an average patient.

   It is not always possible to solicit all levels of an attribute from the decision maker interviews. In these circumstances, the analyst can fill in the gaps afterward by reading the literature or interviewing other experts. The levels specified by the first decision maker are used as markers for placing the remaining levels, so that the levels range from best to worst.  

   The analyst asks the decision maker to rate the relative importance of each level within the attribute using the double-anchored estimation method (Kneppreth et al. 1974).  In this approach, the worst level is assigned a score of zero and the best level is assigned a score of 100.  The decision maker is asked to rate the relative importance of the remaining levels of the attribute.  For example, consider the attribute "Initial licensing fees per physician."  It may have the following levels:

   • No cost (open software)
   • $5000/physician
   • $10,000/physician
   • $20,000/physician
   • $30,000/physician
   • $40,000/physician

   Note that the best and worst costs are assigned by convention a score of 0 and 100.  The following interaction typifies the questioning for the double-anchored estimation method:

   Analyst: Assign 100 to the best level of the attribute "Initial licensing fee" Decision Maker: That will allow me to indicate that the open software is 100 but there are other reasons why I may not want an open software. Analyst Yes, I understand that you have other reasons why you might not get open software.  We are only concerned with licensing fees at this time.  Now assign 0 to the worst level. Decision Maker: That will lead for a rating of 0 for the level "licensing cost more than $40,000 per physician." Analyst What would you rate the level associated with the cost of $30,000 per physician? Decision Maker: That is just as bad as $40,000 per physician, they are both quite expensive.  I will give this a rating of 10. Analyst Where would you  rate $5000 per physician.  Decision Maker: That is really negligible cost.  I rate that around 90 on a scale from 0 to 100, where 100 is no cost at all. Analyst Can you now rate the remaining levels?

Several psychologists have questioned whether decision makers are systematically biased in assessing value. Yates and Jagacinski (1979) showed that using different anchors produced different scores in the double anchored estimation procedure.   It is useful to change the anchors and verify that the ratings assigned by the decision maker are comparable.  By convention, the single attribute value function must range from 0 to 100.  Sometimes, decision makers refuse to assign the 0 value.  In these circumstances, their estimated values should be revised to range from 0 to 100 after the interview.  The following formula shows how to obtain standardized value functions from estimates that do not range between 0 to 100:

For example, if the cost per physician attribute are rated as below:

Then, the maximum value is 90 and the minimum value is 10 and standardized values can be assigned to each level using the formula above.  For example for "Appointment reminder only" the value is:

The resulting standardized values are given as:

4. Assess relative  importance of attributes.    In this step, the analysis proceeds by finding a way for combining single attribute functions into an overall scores representing the vendors desirability across across all attributes. Note that the scoring convention has produced a situation in which the value of each attribute is somewhere between 0 and 100.  The analyst must find an aggregation rule that differentially weights the various attributes.  The most obvious rule is the additive model.  Assume that V represents the value of systems purchased from a vendor.  If the system is described by a series of n attributes of {A₁, A₂, . . . , A_i, . . . , A_n}, then using the additive rule, the overall value equals:

   S = ∑_i W_i V_i(A_j)
   where V_i(A_j) is the value of the j^th level in the i^th attribute, W_i is the importance weight associated with the i^th attribute, and ∑_iW_i = 1.

   Several other models are possible in addition to the additive model. However, the additive form is the most commonly used method of aggregating across multiple attribute evaluations.   The analyst can estimate the weights for an additive value model in a number of ways. This section presents the method of rating the ratio of importance of the attributes first suggested by Edwards in 1977  (Edwards 1977). The attributes are rank ordered, and the least important is assigned 10 points. Then the decision maker is asked to estimate how many times the next attribute is more important. There is no upper limit to the number of points other attributes can be assigned. For example, in estimating the weights for the three attributes (cost, impact on business process, and compatibility with current systems), the analyst and the decision maker might have the following discussion:

   Analyst	Which of the three attributes is most important?
   Decision Maker:	Well, they are all important, but impact on the business process is the reason why we are getting an EHR so I guess I say that is most important.
   Analyst	Which is the least important.
   Decision Maker:	They are all important but among these three I think compatibility with existing systems is least important.  I understand that if we want to have the features that are unique to a system, we may not be able to throw away the old systems and purchase a comprehensive system from the same vendor
   Analyst:	If compatibility with existing system is rated 10 in importance, how many more times is cost more important.
   Decision Maker:	Cost is important.  I would say maybe 3 times more.
   Analyst	That would put cost at 30 if compatibility to existing systems is rated as 10.  Now how many more times is impact on business process more important than cost?
   Decision Maker	I would go with 1.5 times more important.

   In the dialogue above, the analyst first found the order of the attributes, and then asked for the ratio of the weights of the attributes. Knowing the ratio of attributes allowed the analyst to estimate the attribute weights. If the model has only three attributes, the weights for the attributes can be obtained by solving the following three equations: 

   W(Cost)  /  W(Compatibility with existing systems) = 3

   W(Impact on business processes)  /  W(Cost) = 1.5

   W(Compability with existing systems)  +  W(Cost)  +  W(Impact on business processes) = 1

   An easy way to solve these three equations is to assign the least important attribute the weight of 10.  The next attribute will have a weight of 30 and still the next one a weight of 45.  The last equation requires us to divide the estimated weights by sum of all three, yielding the relative weights of 0.12, 0.35 and 0.53.

   Using the attributes in Eden's 2002 survey, we identified the following relative weights for the attributes:

   Attribute	Weight
   The software appeared easy to use	0.17
   Software appeared to improve one or more of the business processes in the practice process.	0.17
   The software provided the most value for cost	0.15
   The software would help the practice perform processes needed to reach our long term business strategy	0.14
   The vendor had many sites and was responsive to our needs during the selection process	0.12
   There were strong testimonies from prior users	0.10
   The software was already in use by other sites affiliated with this practice	0.09
   Software was compatible with existing practice systems in the practice	0.08

   One characteristic of this estimation method is that its emphasis on the ratio of the importance of the attributes leads to relatively extreme weighting compared to other approaches. Thus, some attributes may be judged critical, and others rather trivial. Other approaches, especially the direct magnitude process (assign a weight between 0 to 100 to each attribute), may judge all attributes as almost equally important. One note of caution:  Some scientists have questioned whether decision makers can describe how they weight attributes. Nisbett and Wilson (1977) argued that directly assessed weight may not reflect a decision maker's true beliefs.  Yi (2004) found that patient's decisions depended on the choice of methods of assessing their preferences.  Other investigators review the literature and find that directly assessing the relative importance of attributes is accurate (John and Edwards 1978, Naglie et al. 1997).  The debate is many years old.   The only way to decide if the directly assessed weights reflect the decision maker's opinions is to look at how the resulting models perform. In a number of applications, value models based on directly assessed weights correlated quite well with the subject's judgments (Fischer 1979). The typical correlation is actually in the upper .80s, which is high compared to most social science correlations. This success confirms the accuracy (perhaps one should say adequacy) of the subjective assessment techniques for constructing value models.

5. Discuss next steps.  The next step is to meet as a group.  The analyst must use the end of the conversation to highlight what will happen next and when.  The analyst should mention that he/she will combine the attributes/criteria identified by all decision makers and at the larger meeting provide a chance for others to revise the list.  The larger meeting will focus on discussions of assumptions various people have about the vendors.  The larger meeting will end with each decision maker individually weighing the relative importance of the attributes in their selection of the vendor.  All of this may seem foreign to the decision maker and should be explained by the analyst so that there is no surprise at the next meeting. 

6. An example of calculations.   Suppose that the one-on-one interview has identified two attributes.  Ease of use is considered to be twice more important than cost.  Suppose further that the decision maker has rated the various systems using the following values:
   

   	Ease of Use	Cost of licenses
   System A	20	50
   System B	50	70
   System C	10	100
   System D	0	60

   
   What is the preferred system based on the data solicited from the decision maker?   We start by calculating the relative importance of each of the attributes.  This is done by assigning the least important attribute, cost of licenses,  the weight of 10.  Since ease of use is twice as important as cost of licenses, it is assigned a weight of 20.  The standardized weight is calculated by dividing the assigned scores with the sum of these scores:

Standardized weight for cost of licenses = 10/30 = .33
Standardized weight for ease of use = 20/30 = .67

Next, we need to standardize the relative rating of various attribute levels so that these ratings range from 0 to 100.  Even though the double anchored method asks decision makers to rate the best level at 100 and the worst level at 0, some may not do so.  For these decision makers it is important to standardize the ratings before proceeding.  This is done for each rating by calculating the how the rating corresponds to percent of change from minimum to maximum ratings in the attribute.  For example, System A's rating of 20 in the attribute "Ease of Use" is re-calculated using 50 as the maximum and 0 as the minimum:

This yields the following set of standardized ratings for the two attribute levels:

Now we are ready to calculate the overall score for each of the systems.   The overall score is the sum of the product of ratings within each attribute and the weight assigned to measure the relative importance of the attribute.  For example for System A, the overall score is given by:

Overall Score for System A =
weight for ease of use(rating of System A for ease of use)+weight for cost of licenses(rating of System A for cost of licenses)

Overall Score for System A =
.67(40)+.33(0) = 26.8

The following shows the overall scores for all systems:

As can be seen, this decision maker prefers System B.  The end product of one-on-one interviews is the collection of the attributes, attribute levels, ratings of the attributes, weights of relative importance of attributes so that the decision maker's preferences are known and documented.

Step 3:  Group Meeting

After interviewing each decision maker individually, the analyst collates the responses of all participants and creates a "straw model" of the decision. A straw model refers to a decision model designed to be redone.  The face-to-face meeting starts with a presentation of the  "straw model" and proceeds with a request for improvements. Constructing the model before the group meeting has three advantages:

1. It increases the accuracy of the group's judgment by ensuring that the group process will not prevent an individual from presenting his/her ideas.
2. The interviews prior to the meeting save the group some working time by collecting members' ideas before the meeting.
3. Because the "straw model" serves as a blue print to guide the group discussion, it saves additional time by partitioning the group task into smaller, more manageable discussions.

In Eden's 2002 survey of practices, the following attributes were identified as important when selecting a vendor:

1. The software appeared easy to use

2. Software appeared to improve one or more of the business processes in the practice process

3. The software provided the most value for cost

4. The software would help the practice perform processes needed to reach our long term business strategy

5. The vendor had many sites and was responsive to practice information needs during the selection process

6. There were strong testimonies from prior users

7. The software was already in use by other sites affiliated with this practice

8. Software was compatible with existing systems in the practice

Note that each of these attributes can be further broken down to other (lower level) attributes.  For example the second attribute "Software appeared to improve business process" might be broken into the following sub-set of attributes:

1. Improved billing process

2. More accurate documents

3. Improved ability to analyze managed care costs

4. Improved scheduling process

5. Improved access to patient information at multiple sites

6. Reduced malpractice costs

7. Improved referral process

8. Reduced time for recording patient information

9. Improved communication

10. Improved documented quality

11. Quicker lab results

The analyst lists the basic structure of the "straw model" (the attributes used in evaluation task, the decision events used in a tree structure, or the clues used in a forecast) on flip charts; one component per chart. The flip charts are spread around the room so any member can see what has been said to date. The group convenes to revise the "straw model". 

If the group is too large to meet in a face-to-face session, the Delphi process can be used at this time.   The Delphi process involves three in-related questionnaires.   In the first questionnaire the attributes in the straw model are sent to the group and are asked to modify the existing list of attributes and give reasons for the change.  In the second round, the modified list and the reasons for the modification are mailed back to the participants and each participant is asked to rate the relative importance of the attributes. Participants are also asked to given their reason for the weights they have assigned. Again, the responses are collated and the average weight and the reasons for the modification of the weights are sent back to the participants.  In the last round, the analyst asks the participants to rate the importance of those attributes where there has been disagreement again.  In each round, the Delphi process allows the participants to see the average response from others as well as their reasons.  In this manner, the Delphi process enables the analyst to work on bringing about a consensus among a large number of participants.

When the group is more manageable (less than 15), a face-to-face meeting is organized to review the straw model.  To discourage cliques, all participants sit on one side of the table facing the analyst and the flip charts on the wall.  They find it difficult to carry a back and forth discussion with each other and address the facilitator, who tries to create a synthesis of both of their views.

The analyst introduces himself/herself, explains the group's task and agenda, restates the importance of the task, and asks members to introduce themselves.  These introductions are an important part of the group process.  If members are not explicitly introduced, they will do so implicitly through out their deliberations. 

The analyst presents the "straw model," as his/her poor attempt to pull together what he/she had heard from individual meetings.  The introduction to the straw model should be quick so that the focus does not remain on the overall model.  As soon as possible, the analyst asks the group to revise the model, and focuses group's attention on one of the components in the "straw model" as a starting point. The focus on one component at a time is an important way of managing the group's time and conflicts.  As group members suggest new ideas or modifications, the analyst records them on the appropriate pages in front of the group.  Thus, the analyst serves as a secretary to the group making sure that ideas are not lost. Recording the comments reassures the group members that their ideas are being put in front of the group for further consideration.The process continues until the group identifies, discusses and  revises all relevant components in the decision structure.

Through active listening (e.g., nodding, asking for clarification) and recording of group member ideas on the flip charts, the analyst plays the important role of directing the discussion, preventing premature closure of  ideas (Van de Ven and Delbecq, 1971), returning the group to task related activities, distributing the group's time over different aspects of the task,  restraining expression of critical comments during the generation of ideas and always separating people from ideas so that ideas are judged on their own merits.  The analyst should not participate in the content of the discussions; and should not reword what has been said in his/her own terms.

Step 4:  Construct Scoring Procedure

In this step, the analyst helps the group put numbers to the decision components (e.g. assess weights and rate the relative importance of various levels of the same attribute).  The emphasis is  on assessing the relative importance of different attributes as these scores reflect the underlying tradeoff between attributes.  The analyst has already done this task through individual interviews but the change in the straw model and the potential of differences in how group members rate different attributes requires the analyst to repeat this step.  

To prepare for this task, the analyst lists the questions on separate sheets of paper and asks each participant to respond to the questions individually.   For example, if the analyst needs to assess the relative importance of "cost per physician" and "ease of use", then the following question is typed on one sheet of paper:

Which is more important, cost per physician or ease of use?  ______________
If we assign 10 to the least important attribute, how many times more important is the other? ____ times

Every group member receives the same question.  Responses are written on the sheet and the analyst collects the responses.  If the ratings are near each other, the next attribute is rated.  The process is stopped if there are wide differences among the group members. 

While working individually, the group remains in the presence of one another. Seeing each other hard at work helps the group members exert more effort on the task at hand. As the group proceeds, the analyst collects the group's responses and puts the answers on a flip chart. The scores are not listed in the flip chart in a manner that identifies who has said it.  One purpose of this step is to encourage the group to consider the merit of ideas and not who has expressed them.  When there are major differences among the estimates, the analyst asks the group (not any particular person) to stop working individually and explain their reasoning to the group.

Step 5: Group Discussion

The purpose of participatory vendor selection is to help the organization arrive at a course of action through consensus.  It is important to encourage discussion of issues.  But in order to make sure that important issues are discussed, it is essential to avoid discussion of topics on which there are little debate.  When there is differences in numeric ratings, the analyst asks for the reasons behind the decision makers' ratings. This approach to discussion has been shown to reduce conflict among group members.  Disagreements among group members have many different sources. For example, some disagreements are due to fatigue and unclear problem specification. These disagreements are reduced through clarifying the assumptions behind group member's perspectives. Still other disagreements are due to differences in knowledge and experience.  Discussion may reduce these conflicts if group member's succeed in communicating the rationale for their judgments.  Some group members may raise considerations others have missed.  Group members may see flaws in the logic. Other disagreements are due to value differences.  For example, the CIO may emphasize cost more while the Chief Medical Officer may emphasize ease of use.  In this case, better communication may not reduce this type of conflict. IGP reduces conflicts that are due to misconceptions and miscommunications and accepts other conflicts as inherent in the task.  In these situations, close attention to the ranges of attributes and assumptions various people are making can help the discussion.  It is also possible that these differences cannot be resolved, at which point the analyst notes the differences and proceeds.

Although the analyst encourages group members to resolve their differences through discussion, at some point, it is necessary to stop the interaction and mathematically resolve remaining group differences, such as by averaging the estimates from various group members. If major differences remain, it is necessary to report these differences. 

Step 6:  Measure Performance of Information Systems

When the group discussion ends, the analyst has a tool for scoring various vendors.  This tool specifies a set of attributes and their levels that can be used in rating a vendor.  In addition, the group has specified a scoring system (ratings of each level within each attribute and the relative weight of each attribute).  The analyst can use this information to score each vendor. 

To evaluate the performance of the vendor, the first step is to measure the performance of the vendor on each of the attributes.  There are three ways to accomplish this:

1. Expert ratings:  Current users of the system or people familiar with the systems are asked to rate all of the vendors on each of the attribute levels.  A simple way for doing so is to take any two vendors and make a subjective judgment on who is better.  If the vendor is judged better then it receives a point.  The performances of all pairs of vendors are compared on each of the attributes.  The vendor who receives the highest number of points is assigned the score of 100 on the attribute.  The vendor that receives the lowest number of points is assigned the score of 0 on the attribute.  All other vendors are assigned a score in between 0 and 100 that is proportional to their number of points.
2. Standard cases and objective measures:  Vendors are provided standardized cases and asked to use their information systems to enter the relevant data.  The percent of errors, the number of keystrokes before one can enter the data, the time it takes to enter the data and other similar measures are used to objectively measure the attributes specified by the group for evaluating the vendors.  Kilbridge, Welebob and Classen (2005) report a leapfrog methodology based on assessment of vendors based on a standardized set of clinical cases. 
3. Subjective ratings of the vendor selection committee:  Vendors are invited to demonstrate their systems.  Members of the vendor selection committee use the system and watch the operations of the system and rate the performance of the vendors on each attribute/criteria.  Occasionally, the committee bases its ratings on interviewing others who are currently using the same system.

In measuring the performance of the vendors, it is important to use the same procedures for all vendors so that the ratings are comparable.  For example, if one set of members rate the performance of one vendor, the same set should rate the performance of the remaining vendors.  Care should be taken not to introduce additional sources of bias into the ratings.

Once the performance of the vendor on each attribute has been measured, the second step is to score the performance.  For each attribute, the performance of the system is assigned to one level.  The corresponding score for the attribute level is used as the rating for the vendor on that attribute.  The overall score for a vendor is the sum of the product of the ratings assigned to the vendor multiplied by the relative importance of the attributes.

In the last step, the overall score is used to select the best vendor. 

Step 7:  Documentation

The analyst writes a report showing how various vendors did on each attribute and what scores were assigned to them.  This report contains several different topics including the following:

• Why was the meeting convened?
• Methods used to facilitate the meeting and analyze the findings.
• A detailed description of the consensus criteria, attribute levels and associated score
• Performance of vendors on various criteria
•  Conclusions and next steps in the group's task.

The report is distributed to all participants and is available for others in the organization to examine if they want to understand how the group has arrived at its judgment.  A document about the group deliberation is important not only to people who were in the meeting but also to people who were not. Cinokur and Burnstein  (1974) had individual subjects list the persuasiveness of pro and con arguments.  The net balance of persuasiveness of the arguments correlated with attitude change produced by group discussion.  But, more importantly, other individuals not  present in the group discussion, who were exposed to the same arguments, changed  their attitudes in the same way. The work of Cinokur and Burnstein shows that  the information content of group discussion is important in convincing people outside the group.  A well-documented decision model can help convince others.  It provides a documentation of the group's deliberation.

Concluding Remarks

Vendor selection is time consuming and at times frustrating.  After many product demonstrations, reference checks and site visits, the organization may be left more confused than before the start.  This section has provided an overview of a group process that can be used in vendor selection.  This structured process engages a large number of clinicians and managers in the process.  It is structured so that the information collected at different stages from different people is put to appropriate use.  It is also organized to increase internal communication and produce a consensus around the vendor selection decision.  A systematic and uniform approach to vendor selection appeals to the vendors, they understand what they need to do.  It also appeals to the firm because it engages the relevant people in the decision making process.  

Additional Resources

The following additional resources are available to readers of this chapter:

• Slides for building a consensus model for vendor selection Slides►   
• Video lecture on how to construct consensus models You Tube (start)►  You Tube (continued)►
• Slides for how to construct model of one person's preferences Slides►
• Video on how to to construct a model of one person's preferences  You Tube (start)►  You Tube (continued)►
• Video for calculating weights from ratio estimates SWF►
• Detailed example of a participatory vendor selection Read►

References

1. Alemi F , Jackson M, Parren T, Williams L, Cavor B, Llorens S, Mosavel M.  Participation in Teleconference Support Groups: Application to Drug-Using Pregnant Patients.  Journal of Medical Systems 1997,  Vol 21, 2:  pages: 119 - 125.
2. Austin LC, Liker JK, McLeod PL.  Who Controls the Technology in Group Support Systems? Determinants and Consequences  Human-Computer Interaction, Vol. 8, No. 3: pages 217-236. 1993.
3. Black N, Murphy M, Lamping D,  McKee M, Sanderson C, Askham J, Marteau T.  Consensus development  methods: a review of best practice in creating clinical guidelines. J Health Serv Res Policy. 1999 Oct;4(4):236-48.
4. Carney O, McIntosh J, Worth A.   The use of the Nominal Group Technique in research with community  nurses. J Adv Nurs. 1996 May;23(5):1024-9.
5. Cho HK, Turoff M, Hiltz SR.   The Impacts of Delphi Communication Structure on Small and Medium Sized  Asynchronous Groups.  HICSS, 2003
6. Cruse H, Winiarek M, Marshburn J, Clark O, Djulbegovic B. Quality and methods of developing practice guidelines. BMC Health Serv Res. 2002; 2
7. Fjermestad J, Hiltz SR. "An  Analysis of the Effects of Mode of Communication on Group Decision  Making," HICSS, vol. 01, no. 1, p. 17, January 1998.
8. Forsyth DR. Group Dynamics, Wadsworth Publishing; 3rd edition, 1998.
9. Holzworth RJ, Wills CE. Nurses'  judgments regarding seclusion and restraint of psychiatric patients: a  social judgment analysis. Res Nurs Health. 1999 Jun;22(3):189-201.
10. Jacobs RA. Methods for combining experts' probability assessments. Neural Comput. 1995 Sep; 7(5): 867-88.
11. Jones J, Hunter D. Consensus  methods for medical and health services research. BMJ. 1995 Aug  5;311(7001):376-80.
12. McLeod PL.  University of Michigan An Assessment of the Experimental Literature on Electronic Support of Group Work: Results of a Meta-Analysis.  Computer Interaction 1992, Vol. 7, No. 3, Pages 257-280. 
13. Meyer, M. and J. Booker. 1991. Eliciting and Analyzing Expert Judgment: A Practical Guide. Knowledge-Based Systems, Vol. 5 Academic Press, London.
14. Moon RH. Finding diamonds in the  trenches with the nominal group process. Fam Pract Manag. 1999  May;6(5):49-50.
15. Rohrbaugh J. Improving the Quality  of Group Judgment: Social Judgment Analysis and the Nominal Group  Technique. Organiz. Behav. And Human Perform. Vol. 28, no. 2, pp.  272-288. 1981
16. Pagliari C, Grimshaw J.  Impact of group structure and process on multidisciplinary evidence-based guideline development: an observational study. J Eval Clin Pract. 2002 May;8(2):145-53.
17. Rotondi AJ. Assessing a team's problem solving ability: evaluation of the Team Problem Solving Assessment Tool (TPSAT). Health Care Manag Sci. 1999 Dec;2(4):205-14.
18. Ryan M, Scott DA, Reeves C, Bate  A, van Teijlingen ER, Russell EM, Napper M, Robb CM. Eliciting public  preferences for healthcare: a systematic review of techniques. Health  Technol Assess. 2001;5(5):1-186.
19. Straus SG Technology, Group Process, and Group Outcomes: Testing the Connections in Computer-Mediated and Face-to-Face Groups Human-Computer Interaction, Vol. 12, No. 3: pages 227-266. 1997.
20. Van de Ven AH, Delbecq A.   The Effectiveness of Nominal, Delphi, and Interacting Group Decision  Making Process.  - Academy of Management Journal, 1974. 
21. Vennix JAM. Group model-building:  tackling messy problems System Dynamics Review Volume 15, Issue 4, pages 379 - 401, 2000.
22. Woudenberg F, An Evaluation of Delphi  - Technological Forecasting and Social Change, 1991
23. Gustafson DH, Cats-Baril WL, Alemi F. Systems to Support Health Policy Analysis: Theory, Models, and Uses, Health Administration Press: Ann Arbor, Michigan, 1992.
24. Checkland, P. 1981. Systems Thinking, Systems Practice. New York: John Wiley.
25. Dubois, E., J. P. Finance, and A. Van Lamsweerde. 1982. "Towards a Deductive Approach to Information System Specification and Design." In Requirements Engineering Environments, edited by Y. Ohno. Tokyo: Ohmsha.
26. Eden KB.  Selecting information technology for physicians' practices: a cross-sectional study. BMC Med Inform Decis Mak. 2002; 2: 4. Published online 2002 April 5.  More►
27. Ellis, H. C. 1982. "A Refined Model for Definition of System Requirements." Database Journal 12 (3): 2-9
28. Gustafson, D. H., and A. Thesen. 1981. "Are Traditional Information Systems Adequate for Policymakers?" Health Care Management Review (Winter): 51-63
29. Hira, H., and K. Mori. 1982. "Customer- Needs Analysis Procedures: CNAP." In Requirements Engineering Environments, edited by Y. Ohno. Tokyo: Ohmsha.
30. Hogarth, R. 1981. Judgement and Choice. New York: John Wiley.
31. Huysmans, J.H.B.M. 1970. "The Effectiveness of the Cognitive Constraint in Implementing Operations Research Proposals." Management Science17 (1): 92-104
32. IBM Corporation. 1981. Business Systems Planning- Information Systems Planning Guide, Application Manual, GE20-0527-3.
33. King, W. R. 1978. "Strategic Planning for Management Information Systems." MIS Quarterly 2 (1): 27-37.
34. Kilbridge PM, Welebob EM, Classen DC.  Development of the Leapfrog methodology for evaluating hospital implemented inpatient computerized physician order entry systems.  Quality and Safety in Health Care 2006;15:81-84.   More► 
35. Lunderberg, M. 1979. "An Approach for Involving Users in the Specifications of Information Systems." In Formal Models and Practical Tools for Information Systems Design, edited by H.J. Schneider. Amsterdam: North Holland.
36. Mason, R.O., and I. I. Mitroff. 1973. "A Program for Research on Management Information Systems." Management Science 19 (5): 475-87.
37. Mintzberg, H. 1975. "The Manager’s Folklore and Fact." Harvard Business Review 53 (4): 49-61.
38. Mitroff, I., and T. Featheringham. 1974. "On Systematic Problem Solving and Error of the Third Kind." Behavioral Science 19 (3): 383-93.
39. Munro, M. C., and G. B. Davis. 1977. "Determining Management Information Needs- A Comparison of+ Methods." MIS Quarterly 1 (2): 55-67.
40. Rockart, J. F. 1979. "Critical Success Factors." Harvard Business Review 57 (2): 81-91.
41. Ross, D. T., and K. E. Schoman. 1977. "Structured Analysis for Requirements Definition." IEEE Transactions on Software Engineering SE-3 (1): 6-15.
42. Taggart, W.M., and M. O. Tharp. 1977. "A Survey of Information Requirements Analysis Techniques." Computing Surveys 9 (4): 273-90.
43. Valusek, J. R. 1985. "Information Requirements Determination: An Empirical Investigation of Obstacles Within an Individual." Unpublished doctoral dissertation, University of Wisconsin-Madison.
44. Wack, P. 1985. "Scenarios: Uncharted Waters Ahead. "Harvard Business Review 6363 (5): 72-89.
45. Yadav, S. B. 1983. "Determining an Organization’s Information Requirements: A State-of-the-Art Survey." Data Base 14 (3): 3-20.
46. Doolan DF, Bates DW, James BC. The use of computers for clinical care: a case series of advanced U.S. sites J Am Med Inform Assoc 2003;10:(1):94-107.