Performance measurement can be an important strategy to help improve clinical care. And performance measurement is an area that is fraught with pitfalls which have the potential to undermine quality, penalize quality performers and reward others who are not providing quality care. We offer practical help.

 Web Exclusive: Written by Michael Stuart and Sheri Ann Strite and reprinted here with permission.

Pitfalls to performance measurement

When drawing conclusions from data, watch out for conflicting measures

Story originally published January 23, 2006

Regulators, payers, patients and others understandably want to determine which healthcare providers deliver the highest quality of care. Interest in performance measurement has increased since such groups such as the Institute of Medicine, the Center for the Evaluative Clinical Sciences at Dartmouth, the CMS, and the Rand Corp. have reported estimates that 20% to 50% of all prescriptions, visits, procedures and hospitalizations in the U.S. may be inappropriate. That translates into patient deaths and injury and the waste of an estimated $400 billion annually.

Clearly there is a need to improve healthcare, and performance measurement is an increasingly applied component of quality improvement. Yet there are pitfalls to performance measurement of which many users are unaware.

We offer an outline of some important limitations we believe healthcare leaders and managers should consider as they attempt to improve quality through the use of performance measures within their organizations. Providers must distinguish between three main uses of performance measures: required reporting, quality improvement, and evaluating organizational or individual performance. Each of these uses requires a different approach because of the possibility of invalid measures. Providers must understand the limitations and pitfalls of drawing conclusions from performance measurement because of the problems resulting from conflicting measures, which can result in decreased quality, waste and incorrect conclusions about individual health organizations or practitioners.

Components of a measure
A good performance measure consists of a valid numerator and denominator, frequency of occurrence and the data-gathering process. It is often expressed as a percentage or a rate. The text statement for a diabetic patient-care performance measure might be, "The percent of patients with a diagnosis of diabetes mellitus receiving at least one hemoglobin A1c annually."

A valid denominator means that it specifies the right base from which the measurement will be made. The denominator must have appropriate criteria for the pool of whom or what is eligible for measurement. For example, a denominator measuring clinical improvement for cervical Pap smears that does not exclude women without a cervix would be invalid.

Numerators generally count events such as something that happens to a patient or something patients receive - typically, this is an outcome, an intervention, a service or a process. Examples of numerators are the number of patients with diabetes mellitus from the denominator (the population eligible for measurement) who have received an annual hemoglobin A1c, or the percent of charts available for patient appointments at an outpatient clinic.

A key point -- often misunderstood -- is that numerators should be based on valid, useful and usable scientific evidence. For health status outcomes and interventions, this requires that we know what interventions are likely to result in improvement. With rare exceptions, only well-designed and conducted randomized controlled trials can demonstrate cause and effect relationships, and only valid and useful information from trials data should be used for interventions.

Medical leaders, administrators and others should be aware of the significant potential for selection bias, observation bias, confounding and the play of chance when relying on observational data -- and performance measures are one form of observational data. These threats to validity can confound performance comparisons between institutions, units and individuals. Trying to resolve this by adjusting for case mix is analogous to using models to adjust for patient differences in observational studies dealing with therapy -- the potential to be misled by confounding factors remains high. Databases and clinical records are useful for measuring processes, but are not reliable for attempting to "prove" that a health status improvement was the result of an intervention. Database information, observational studies and opinions of experts can inadvertently mislead.

Unfortunately, because of a general lack of knowledge of these potential problems, providers may be required to report outcomes to stakeholders even when the performance measure or the organization's outcomes lack validity.

Problematic results
Feedback to a clinician in the form of a performance report may be of great value as a way of encouraging his or her participation in quality improvement efforts and focusing attention on improving processes of care and attention to patients' needs. However, because of significant validity and reliability problems inherent in observational data, it is altogether a different issue when an individual provider's performance is made available to others in the form of a performance "report card" or when an individual's income is based on a limited set of performance measures.

Here are several examples of problems that can result:

• The wrong denominator: A colon cancer screening quality improvement project at the Veterans Administration Hospital in San Francisco resulted in the facility failing to meet a national target and the hospital faced financial penalties. However, an audit revealed that 47% of the patients included in the measure had declined screening, 12% failed to make their appointments for screening, 11% had chart documentation that screening was not indicated and 42% of the counted patients received diagnostic testing rather than screening (i.e., they had signs or symptoms of disease). Thus, the conclusion that the hospital was failing to meet national VA benchmarks was incorrect.

• The wrong numerator: Some groups recommend routine screening of all newborns for hearing problems during postpartum hospitalization-this is even required by law in many states. There is, however, insufficient evidence to conclude that such testing leads to improved speech and language skills at 3 years of age. It is also unclear from the best available evidence if potential benefits outweigh potential harms of false-positive tests. Unfortunately, many stakeholders demand performance data on this measure. When looking at internal quality projects it would be better to select a more valid measure.

• Problems in judging the quality of a clinician: A physician may take appropriate actions to improve quality of care, but because of patient factors, systems factors or small sample size, the physician's performance may not result in clinical improvement.

Examining the use of profiling family physicians for glycemic control in their diabetic patients is instructive. It has been reported that in a typical family practice, only 4% or less of variance in hospitalization rates, visit rates, lab utilization and glycemic control in diabetics can be attributed to differences in physician practice patterns. For profiles of glycemic control, outlier physicians could dramatically improve their profiles by pruning their panels of as few as one to three patients with the highest HbA1cs levels. This gaming of the system could not be prevented by case-mix adjustment.

Leaders and managers should remember that conclusions about individuals -- and even organizations -- based on performance measures alone should be drawn with great care. A key issue is how much of the outcome is due to selection bias, sample size, and other factors such as how much of the outcome is really under the control of the clinician or the health system and how much is not.

Performance measurement is an important component of quality improvement efforts in healthcare. However, if performance measures are not designed and used correctly, they are rendered statistically or clinically meaningless -- a waste of resources and a threat to quality. To bring clinicians on board with quality efforts and to improve patient safety and outcomes, we must ensure that each measure captures the data intended and is used appropriately.

Michael Stuart is president of the Delfini Group and clinical assistant professor in the Department of Family Medicine, University of Washington School of Medicine, Seattle. Sheri Ann Strite is a principal and managing partner in the Delfini Group, Portland, Ore.