Volume — Use of Evidence:
Applying the Evidence

newest
01/06/2011:
"How to Evaluate Medical Science," Letter to the Editor by Sheri Ann Strite & Michael E. Stuart MD,
The Atlantic, January/February 2011

Dec 2010: Gastro-esophageal Reflux Disease (GERD), The Purple Pill and Plenty of Pennies

Contents      

• Delfini Commentary on "Lies, Damned Lies, and Medical Science," by David H. Freedman, The Atlantic, November 2010
  • Link to the Article is available above as is our Letter to the Editor and our Full Commentary
• Guidelines & Effectiveness of Implementation
• Oregon Preferred Drug List
• Successful Evidence-based QI Project: Diabetes Management at Dreyer Medical Clinical
• External Validity: Case of the Carotid Stent
• Success Story at the Delfini Project Showcase: Nephrology Clinical Practice Guideline for Primary Care
• Also see On the Same Page™ for applying the evidence through patient-centered resources

Go to DelfiniClick™ for all volumes.

"Lies, Damned Lies, and Medical Science," by David H. Freedman, The Atlantic, November 2010
"How to Evaluate Medical Science," Letter to the Editor by Sheri Ann Strite & Michael E. Stuart MD,
The Atlantic, January/February 2011

• The Article: Lies, Damned Lies, and Medical Science
• Our Letter to the Editor as Published in The Atlantic: How to Evaluate Medical Science (search for Delfini and scroll up or read directly below on our website)
• Our Full Commentary

How to Evaluate Medical Science
Letter to the Editor by Sheri Ann Strite & Michael E. Stuart MD, The Atlantic, January/February 2011

We applaud David H. Freedman’s “Lies, Damned Lies, and Medical Science” (November Atlantic), having long been admirers of professor John Ioannidis. We too evaluate medical evidence and train physicians and others in how to analyze studies for reliability and clinical usefulness. However, we believe the problem is larger, and the consequences of applying the results of misleading science more deleterious, than implied.

Full Delfini Commentary on "Lies, Damned Lies, and Medical Science," by David H. Freedman, The Atlantic, November 2010

We have long been admirers of Professor John Ioannidis and routinely cite his important PLoS article, “Why Most Published Research Findings are False,” in our training programs.[1] We were excited to see him featured in The Atlantic in David Freedman’s important article, "Lies, Damned Lies, and Medical Science," which you can read here.

One of the main points in the article is that Professor Ioannidis “…charges that as much as 90 percent of the published medical information that doctors rely on is flawed.” We are hopeful that this important piece casts a spotlight on the problem of flawed and potentially misleading medical science that we continue to raise wherever and however we can.

However—and importantly—we believe the problem of the creating and using of potentially misleading science to be much, much larger than expressed in the article, and we believe the consequences to the public are much, much greater than implied. It is our experience having reviewed hundreds of clinical trials that roughly 90 percent of randomized controlled trials—forget the observations!—are not only flawed, but are so flawed that they are not reliable or clinically useful. So the reality is even worse, in our estimation. And the documentation of resulting patient harms is significant and significantly underreported.

Adding to this problem is that the vast majority of physicians, clinical pharmacists and other healthcare professionals have no idea how to evaluate a study in even a basic way (informally, roughly over 70 percent of doctors fail our simple 3-question critical appraisal pre-test). If they have even basic skills, we believe they would be able to dodge maybe as high as 75 percent of potentially misleading clinical trial results, or more. Care for patients would be much better and waste much reduced. (Major groups have estimated that somewhere in the neighborhood of 20 to 50 percent of care in the US is inappropriate.[2-6] That translates into at least one hundred billion dollars each year.)

Ironically, it is relatively easy to learn basic critical appraisal skills. Yet, schools generally do not provide effective training in how to conduct studies that are likely to yield valid and useful science and—possibly more importantly—in how to critically appraise the medical literature and apply it. A big issue is that it is from this untrained pool that many of our researchers come, not to mention editors and peer-reviewers (and future academicians). In writing the World Association of Medical Editors in 2007 about their lack of critical appraisal criteria, we received the following response: “Thanks for your interest…regarding the issue of inaccuracy and error in the medical literature (and in fact in all scientific literature). You are correct that this is a large problem. No doubt it exists partly because many reviewers, editors, and authors lack skill, training, and understanding in these issues.”[7]

If healthcare professionals are not competent in evaluating primary studies (original studies such as RCTs) they will not be able to determine if study results are reliable. The problem is carried further: when they read secondary studies (systematic reviews and meta-analyses) without critical appraisal skills, they will be unable to determine if the included primary studies are reliable. And they will have the same problem evaluating secondary sources (guidelines and other derivative works such as monographs and textbooks).

The entire culture of medicine and healthcare needs to change to one in which effective evidence-based practice is the norm with a true understanding of what that really means. This will not be easy because, unfortunately, the magnitude and severity of the problem is not widely appreciated and frequently the concept of evidence-based practice is misunderstood. It all starts with our schools. This article casts light on the tip of a very big and dangerous iceberg, and we are thankful to Dave Freedman for writing it and for Professor Ioannidis for all his important work including participating with Dave in this story.

References

1. Ioannidis JPA. Why Most Published Research Findings are False. PLoS Med 2005; 2(8):696-701. PMID 17593900.

2. Chassin MR, Galvin RW (1998) The National Roundtable on Health Care Quality. The urgent need to improve health care quality: Institute of Medicine National Roundtable on Health Care Quality [consensus statement]. JAMA 280:1000–1005.

3. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J et al. (2003) The quality of health care delivered to adults in the United. N Engl J Med 348:2635-45.

4. Kerr EA, McGlynn EA, Adams J, Keesey J, Asch SM (2001) Profiling the quality of care
in twelve communities: results from the CQI study. Health Aff (Millwood) 23:247-56.

5. Skinner J, Fisher ES, Wennberg JE (2001) For the National Bureau of Economic Research. The efficiency of Medicare. Working Paper No. 8395. Cambridge, MA: National Bureau of Economic Research. Available: papers.ssrn.com/sol3/papers.cfm?abstract_id=277305. Accessed 13 January 2010.

6. Centers for Medicare and Medicaid Services, Office of the Actuary, National Health Statistics Group, National Health Care Expenditures Data, January 2010.

7. Personal correspondence from WAME, World Association of Medical Editors.

Gastro-esophageal Reflux Disease (GERD), The Purple Pill and Plenty of Pennies
12/20/2010

We thought we would pass on some information about PPIs, having been asked by a medical director friend of our what we thought about Nexium®—aka the controversy concerning the purple pill (esomeprazole) versus Prilosec (omeprazole) when considering comparative efficacy and cost. Our friend mentioned that he was concerned because Nexium costs about $5.60 per pill and Prilosec costs about 60 cents per pill, and some of his physicians believe that Nexium is more effective for GERD (gastro-esophageal reflux disease) than Prilosec.

Background: In the 1980s, proton pump inhibitors (PPIs) were quickly adopted for reducing symptoms associated with gastric hyperacidity, e.g., heartburn and reflux symptoms. Priolosec® (omeprazole) was to go off patent in 2001, and generic omeprazole would become available at a lower cost.

In 2000 a trial, “Esomeprazole improves healing and symptom resolution as compared to omeprazole in reflux oesophagitis: a randomized controlled trial [1],” concluded that, “Esomeprazole was more effective than omeprazole in healing and symptom resolution in GERD patients with reflux oesophagitis, and had a tolerability profile comparable to that of omeprazole.” In this trial, 1960 patients with endoscopy-confirmed reflux esophagitis were randomized to once daily esomeprazole 40 mg (n= 654) or 20 mg (n= 656), or omeprazole 20 mg (n=650) for up to 8 weeks.

What We Found: This study had a few flaws. For example, details of randomization were not reported and baseline characteristics differed with more patients with mild (versus more severe) esophagitis in the esomeprazole 40 mg group than in the omeprazole 20 mg group. Four of the 9 authors were from Astra Zeneca, and the study was supported by a grant from Astra Zeneca. (We do not reject studies because of industry involvement, but we do pay attention to this.)

Three subsequent trials using the same doses of the two drugs were also conducted, and two of the three trials reported statistically significant differences in symptom relief (favoring esomeprazole) with a pooled estimate of effect of an 8%, 95% CI (3% to 13%) difference in the proportion of subjects achieving resolution of GERD symptoms.

This and other information found in a very helpful recent OHSU review of PPIs paint a more complete picture of the comparative efficacy of omeprazole compared to esomeprazole (and other PPIs) [2]:

• Among 16 head-to-head trials of PPIs, those with comparable doses did not find differences in symptom relief or healing of esophagitis between the two drugs:
  • Pooled rate for resolution of symptoms at 4 weeks in 7 trials with esomeprazole 40mg was 73%, 95% CI (65% to 82%).
  • Pooled rate for resolution of symptoms at 4 weeks in 2 trials with omeprazole 40 mg was 76%, 95% CI (65% to 87%).
  • Pooled rate for esophagitis healing in 8 trials with esomeprazole 40mg was 78%, 95% CI (73% to 83%). At 8 weeks the rate was 90%, 95% CI (88% to 92%).
  • Pooled rate for esophagitis healing in 2 trials with omeprazole 40 mg was 68%, 95% CI (59% to 78%). At 8 weeks, the rate (1 trial) was 87%, 95% CI (76% to 99%).

“Is esomeprazole really clinically superior to omeprazole in reducing symptoms in patients with symptoms of GERD (gastro-esophageal reflux disease or healing rates in esophagitis?”

Our conclusion: There is insufficient evidence to conclude that there are meaningful clinical differences between omeprazole (Prilosec) and esomeprazole (Nexium) when given in comparable doses for symptom relief or healing of heartburn or esophagitis. Resolution of symptoms and healing rates are very high with either drug—in the neighborhood of 75 percent. The cost differences, however, are significant.

References
1. Kahrilas PJ, Falk GW, Johnson DA, Schmitt C, Collins DW, Whipple J, D'Amico D, Hamelin B, Joelsson B. Esomeprazole improves healing and symptom resolution as compared with omeprazole in reflux oesophagitis patients: a randomized controlled trial. The Esomeprazole Study Investigators. Aliment Pharmacol Ther. 2000 Oct;14(10):1249-58. PMID: 11012468

2. The most useful systematic review we could find comes from Oregon Health Sciences University (OHSU) Center for Evidence-based Policy: (http://derp.ohsu.edu/about/final-document-display.cfm). Accessed 7/13/10.

Guidelines & Effectiveness of Implementation

Tremendous effort goes into the development of high quality clinical guidelines and effective disease management programs. But, there is much uncertainty about how to effectively implement the final product — the evidence-based clinical improvements. In this systematic review, Weingarten et al report that clinicians do change their practice based on provider education efforts, among other strategies. Click to learn more about improvements in care directed at clinicians and patients.

Link to the Abstract (Full Text Available)

BMJ 2002;325:925 ( 26 October )

Weingarten S. et al. Interventions used in disease management programmes for patients with chronic illness — which ones work? Meta-analysis of published reports.

http://bmj.com/cgi/content/abstract/325/7370/925

Oregon Preferred Drug List: Conference Report

"Adding Value to the Cost Equation with Prescription Drugs: The Oregon Experience" — Tuesday, February 25th, Seattle Washington

Mike was on a panel, including many notables. Keynote address was by John Kitzhaber, MD, Former Governor, State of Oregon. The conference was a great success. Read about the program at —

To access, copy the following into an internet search engine:
http://seattletimes.nwsource.com/cgi-bin/ PrintStory.pl?document_id=134641620&
zsection_id=268448406&slug= prescription26m&date=20030226

Successful Evidence-based QI Project: Diabetes Management at Dreyer Medical Clinical
Example provided by Rami Rihani, PharmD, Director of Pharmacy

Delfini Introduction
Measuring clinical improvements is complex. One of the most important, frequently misunderstood issues is that cause and effect relationships can only be drawn with reasonable certainty from valid experiments (RCTs). However, if we have valid evidence from RCTs that an intervention leads to improved clinical outcomes, it is then reasonable to use process measures to evaluate the success of our evidence-based clinical improvement project.

Generally, we advise people to measure — not health status outcomes — but to perform a process measurement to evaluate the success of application of the intervention. In other words, we advise people to measure the success of implementation of the clinical improvement. For example, if we are trying to ensure patients get a beta-blocker post-MI, we would recommend looking to see if prescriptions increased for hospitalized MI patients — not to measure whether patient survival was improved. This is because observational data, such as information extracted from databases, can be highly prone to confounding. If health status outcomes are measured, then we advise people to ensure that there is a sufficient understanding of all those utilizing the data that conclusions drawn from observational data can be misleading. In the above example, if patient survival decreased, there could be many explanations.

However, if a health status outcome is measured, and if the before/after change is dramatic, it is reasonable to hypothesize that our project has been successful. For example…

Problem
Many diabetics have difficulty achieving a HbA1c <7.0. Frequently diabetics are told their HbA1cs are too high but active medication change is not aggressively pursued.

Evidence-based QI Project: A quality improvement group at Dreyer Medical Clinic developed a disease management initiative using PharmDs to actively titrate dosages of insulin and other drugs based on the Intermountain Health Care (IHC) diabetes management protocol. The process is as follows:

• Primary care physician (PCP) refers patient to the diabetes management program;
• PharmD aggressively titrates medication based on IHC protocol;
• PharmD monitors for safety and efficacy of medication interventions in collaboration with the PCP

Outcomes

Delfini Commentary
There was a significant improvement in the percent of patients achieving goal HbA1c and LDL associated with this project.

It is reasonable to believe that the clinical improvement project was successful. Using outcomes data from the UK Prospective Diabetes Study 35 (1), the QI team estimates that since inception, the disease management initiative resulted in the prevention of —

• four diabetes related deaths and
• nine microvascular events (defined as renal failure, death from renal failure, retinal photocoagulation, or vitreous hemorrhage)

1. Stratton, I,M., Adler, A.I., et al, Association of glycaemia with macrovascular and microvascular complications of type 2 diabetes (UKPDS 35): prospective observation study. BMJ 2000; 321; 405-12.

The lead article in the Oct. 7, 2004 issue of The New England Journal of Medicine, (Yadav JS, Wholey MH, Kuntz RE, et al. Protected Carotid-Artery Stenting versus Endarterectomy in High-Risk Patients N Engl J Med 2004;351:1493-501 — PMID: 15470212 — abstract), known as the SAPPHIRE trial, is an RCT with what appears to be adequate randomization/concealment of allocation, similar baseline characteristics of subjects, adequate blinding and with intention to treat analysis comparing carotid stenting using an embolic protection device to endarterectomy in patients with symptomatic carotid stenosis of at least 50 percent or asymptomatic stenosis of at least 80 percent and at least one high risk factor such as recurrent stenosis after endarterectomy e.g., age>80, cardiac disease or other significant medical problems. More than 20% of the patients in both treatment groups were patients with recurrent stenosis after endarterectomy.

The primary end point was a composite of death, stroke or MI within 30 days after the intervention, or death or ipsilateral stroke between 31 days and 1 year. The study appeared to be a sufficiently powered equivalence study. Study team members included both a surgeon and an interventionist who could prevent randomization of enrolled subjects if, in their judgment, the procedures could not be performed safely in these high risk patients. 747 patients were enrolled and 334 underwent randomization. The primary endpoint occurred in 20 patients in the stenting group (12.2 percent) and in 32 in the endarterectomy group (20.1%), yielding a P value of 0.004 for non-inferiority. At one year, revascularization was repeated in 0.6 percent of the stent patients and 4.3 percent of the endarterectomy patients; P=0.04. The authors conclude that in patients with severe carotid-artery stenosis and coexisting conditions, carotid stenting with an emboli protection device is not inferior to carotid endarterectomy.

Question
What are the biggest threats to validity in this study (based on the above information)?

Our Answer
There is a huge problem with selection bias and external validity in this study. More than 20% of the patients in both treatment groups were patients with recurrent stenosis after endarterectomy which may bias the results towards stenting. 55% of enrolled study patients were excluded by the physician team raising further concerns about selection bias and issues surrounding highly selected patients.