Full text is available to subscribers and can be obtained for a fee for non-subscribers at http://www.healthaffairs.org/

Several years ago, we received a phone call from a medical director asking if we would do a limited review of radiofrequency for the treatment of gastro-esophageal reflux disease (GERD in the US; GORD in UK). He had received a request for coverage of this procedure by a physician and was concerned that there was not good scientific evidence supporting use of this procedure. His health plan policies required proof of efficacy.

We agreed. We contacted the requesting physician and asked him to recommend his best three studies for review. (He pleaded for more - we relented and said, “Okay, four.”) Of the 4 studies he sent us, only 1 was a randomized controlled trial (RCT), and so we had to surmise that if he had been given permission to send us even more, those additional studies, too, would have been observational.

Observational studies should not be used for addressing questions of efficacy of interventions with rare exception.[1,2] The only exception that is thus far agreed upon is when all-or-none results are achieved, and this is extremely rare.

And one RCT generally will not be considered sufficient to declare “proof.” Even a meticulously designed, executed and reported RCT may still result in findings merely due to chance, not to mention that only through replicated studies might one discern the unfortunate occasional fraud. And this RCT could not even come close to being a candidate for definitive proof of anything. While thoughtfully designed by investigators more knowledgeable of research design than many we have seen, it was very small and had myriad problems as our critique — reproduced below — demonstrates.

Even the authors - very responsibly - themselves acknowledged that their study should only be considered preliminary and to have proved nothing (a point that the requesting physician seemed to have overlooked in his desire for a solution for this problem that he, himself, believed in).

And now, in The Lancet, Mehran Anvari[3] of St Joseph’s Healthcare, Hamilton ON, Canada, comments upon a recently published review which considers clinical trials from the past three decades on the use of endoluminal therapies for GERD/GORD.[4]

Anvari writes that the systematic reviewers found that “…no study of any techniques has provided adequate ‘scientific and clinical data on their safety, efficacy and durability to support the use of endoluminal therapies for GORD [gastro-oesophageal reflux disease] in routine clinical practice.’ Thus we could be left with a sense of pessimism about the future use of endoluminal therapies for this common disease, at a time when similar endoluminal approaches are being proposed for other common disorders, such as morbid obesity.”

He tells us that the reviewers, “…also highlight the surprising lack of well-designed randomised trials, even though many of the endoluminal therapies have been in clinical use for several years and many of the trials of these techniques have come from respected academic centres.” He goes on to say, “It almost seems that the standards established for the assessment of treatment efficacy in the control of GORD… were lowered [emphasis ours] when it came to assessing these approaches.”

He also presents the sobering news that, “As a result, many patients have undergone procedures that have since been abandoned and proven ineffective, and an unfortunate minority has had substantial side-effects or even died.”[5-9]

Indeed. And how many patients may have been told that research studies have shown benefits of this treatment? Isn’t it possible that some patients would not have elected to undergo this procedure if they had been told, what Anvari states — most correctly — in his commentary, “…up to this point, there is no clear evidence to support the use of any of the endoluminal therapies…in routine clinical practice.”

This is what we advocate:

While some would say if there is no evidence of benefit, don’t do it, we, personally, do not wish to tell others how to practice medicine. That is not what we see our role to be. No evidence of benefit is not the same thing as evidence of no benefit. And it is possible that something for which there is no evidence works. (In fact, some topics are so inherently difficult to study that we will never have good evidence — and this doesn’t mean that practitioners will stop trying to find solutions and offering options ot patients.)

We do, of course, advocate that a science-based approach be taken in all possible instances. It is important to try and study — correctly — cause and effect for interventions. It is important to report findings responsibly. It is important that all users of such information have skills to evaluate science and that they do so.

Know and clearly state the evidence first! Then decisions can be made taking into account other factors.

However, it is our strong belief that patients should be provided with honest information to enable them to make choices that are in their best interest in accordance with their values and preferences. This requires that all working in health care decision-making acquire the appropriate skills. There are many, many resources available to help achieve this — including information we make freely available on our site from our own work and the work of others.

Our patients deserve this — and indeed — their lives depend upon it. And that is why we are doing this work.

References
1. Delfini Primer: Problems with Case Series

2. Delfini Primer: Problems with the Use of Observational Studies to Draw Cause and Effect Conclusions About Interventions

3. Anvari, M. Endoscopic treatments for gastro-oesophageal reflux disease. [Comment] The Lancet. Vol 371. March 22, 2008. 965-966

4. Fry LC, Mönkemüller K, Malfertheiner P. Systematic review. Endoluminal therapy for gastro-oesophageal refl ux disease: evidence from clinical trials. Eur J Gastroenterol Hepatol 2007; 19: 1125–39.

5. Pleskow D, Rothstein R, Lo S, et al. Endoscopic full-thickness plication for the treatment of GERD: a multicenter trial. Gastrointest Endosc 2004; 59: 163–71.

6. Noh KW, Loeb DS, Stockland A, Achem SR. Pneumomediastinum following enteryx injection for the treatment of gastroesophageal refl ux disease. Am J Gastroenterol 2005; 100: 723–26.

7. Wong RF, Davis TV, Peterson KA. Complications involving the mediastinum after injection of Enteryx for GERD. Gastrointest Endosc 2005; 61: 753–56.

8. Tintillier M, Chaput A, Kirch L, Martinet JP, Pochet JM, Cuvelier C. Esophageal abscess complicating endoscopic treatment of refractory gastroesophageal refl ux disease by enteryx injection: a fi rst case report. Am J Gastroenterol 2004; 99: 1856–58.

9. Tuebergen D, Rijcken E, Senninger N. Esophageal perforation as a complication of EndoCinch endoluminal gastroplication. Endoscopy 2004; 36: 663–65.

10. Ebell M. An introduction to information mastery, July 15, 1988. http://www.poems.msu.edu/InfoMastery/default.htm. Accessed December 21, 2007.

For more than two decades, observers like Jack Wennberg have pointed out geographic variations in health care practices and recommendations in the United States. David Eddy has elegantly demonstrated extreme variation in experts’ recommendations for health care interventions. And yet, many clinicians rely on expert opinion as their primary source of information. EBM emphasizes the need to critically appraise information sources for validity and clinical relevance as a way of dealing with such variation.

Recently Schaafsma FG, Verbeek JH, Hulshof CT, van Dijk FJ have reported on how expert opinion differs from information gathered from an evidence-based approach (Caution required when relying on a colleague's advice; a comparison between professional advice and evidence from the literature. BMC Health Serv Res. 2005 Aug 31;5(1):59. PMID: 16131405)

The authors presented 12 occupational medicine cases to 14 occupational medicine specialists from differing geographical areas in the Netherlands. The authors also searched the literature and critically appraised the evidence on the 12 problems. Questions regarding management were stated so that specialists could answer with yes/no responses. [Example: For a 36-year old caretaker at a secondary school with a lateral ankle ligament rupture treated with tape for three weeks, is it safe to resume work? Evidence-based answer: Yes]

The authors found that 53% (95% confidence interval 42% to 65%) of all professional advice was not in line with the research literature. In 18 cases (24%), professional advice explicitly referred to up-to-date research literature as their source of information. These cases were substantially less incorrect (17%) than advice that had not mentioned the literature as a source (65%). The authors conclude that less than half of the given professional advice by experts to a practical occupational health problem was in line with evidence from the research literature.

This article points out the need for caution in accepting information from “experts” without evaluating the actual evidence from the published medical literature.

Newer EBM texts, when addressing the areas of diagnosis and screening, frequently state that diagnostic tests that produce large changes from pretest to post-test probabilities are very useful clinically.

The teaching goes like this: Likelihood ratios can be utilized along with practitioners’ clinical estimate of the pretest probability of disease, incorporating the results of the diagnostic tests and then deriving an individual patient’s probability of having a disease or condition.

Phelps and Levitt point out (Acad Emerg Med. 2004 Jun;11(6):692-4.) that this approach to generating post-test disease probabilities has never been validated and that the approach may result in very inconsistent results between clinicians.

In a cross-sectional cohort study of emergency and internal medicine residents and faculty, the authors presented clinical vignettes and asked clinicians to estimate the likelihood of common diseases based on history and physical exam findings. No lab or imaging results were provided. They reported that the mean pretest probability estimates of disease ranged from 42% (95% confidence interval [95% CI] = 36.6% to 47.4%) to 77% (95% CI = 72.9% to 81.1%). The smallest difference in pretest probability magnitude for a single vignette was 70% (range 30-100%; interquartile range [IQR] 64-80%), whereas the largest was 95% (range 3-98%; IQR 30-60%).

Their conclusions were that there is wide variability in clinicians’ estimates of pretest probability of disease and that post-test disease estimates are also likely to be inconsistent.

Because of such wide variability, we prefer to concentrate on more traditional statistics derived from the classical 2x2 table such as sensitivity, specificity, positive predictive value and negative predictive value and skip the hassle of estimating pretest probability and using nomograms or tables to generate post-test probabilities.

The abstract of this article can be found at:

One of the main reasons for using valid, relevant evidence in health care is to more accurately predict outcomes from various interventions and thus be equipped to make informed choices. For example, it is very useful to know the evidence for outcomes by risk category for cardiovascular disease. The higher an individual’s risk for a cardiovascular event, the greater the benefit from interventions such as lipid-lowering with statins.

Why would such a small percentage of elderly patients with CV disease or diabetes be prescribed lipid-lowering drugs? Why would the likelihood that physicians prescribe statin therapy be inversely correlated with CV risk?

Comment: This is yet one more study demonstrating the huge problem of underuse of proven interventions in health care. However, this is apparently the first article pointing out the treatment-risk paradox and is worth reading. The authors suggest that the treatment risk paradox in these elderly patients might be explained by:

Pharmacy and Therapeutics committees and clinicians are frequently faced with issues regarding substitution for agents in a specific drug class. CD Furburg and BM Psaty, using ace-inhibitors (ACEIs) as an example, make a number of interesting points in an article entitled, “Should evidence-based proof of drug efficacy be extrapolated to a “class of agents”? (Circulation 2003;108:2608-2510). Below is a summary of some key points made by the authors of the article and also some points made by the editorialists, EM Antman and JJ Ferguson (Circulation 2003;108:2604-2607).

Generic substitution is the act of dispensing a different brand or an unbranded drug product that is the same chemical entity and meets US Food and Drug Administration criteria for bioequivalence (e.g., generic lisinopril in place of brandname Prinivil [Merck & Co.] or Zestril [Zeneca Pharmaceuticals]).

Therapeutic substitution is the dispensing of an alternate chemical entity for the original drug prescribed by the physician from the same general therapeutic class (e.g., a physician orders enoxaparin, but the hospital pharmacy dispenses dalteparin; a physician writes a prescription for ramipril, and the pharmacy dispenses moexipril).

The HOPE trial, using ramipril 10mg, reported the following relative risk reductions in patients with vascular disease or diabetes and one additional risk factor:

However, the evidence is less robust for some of the other ACEIs:

Does it make sense to declare all of the ACEIs equivalent? What about differing doses that have not been tested?

Furburg and Psaty point out that all 7 drugs have FDA approval for treating hypertension, heart failure and LV dysfunction. All 7 have been shown to lower BP and improve hemodynamic measurements. However, only 4 of these drugs—enalapril, captopril, ramipril and trandolopril—have been shown to improve long term survival or clinical outcomes compared to placebo in large, RCTs.

They point out that:

Given all of the above, the authors state that efforts should be made to use drugs with the best-documented outcomes and in full doses. Further, the use of me-too drugs without adequate documentation should be restricted. Comparability should be established in trials where two agents in a class are compared.