Quality of Evidence:
Secondary Sources

newest
03/23/08: Another Cautionary Tale About Clinical Guidelines »

Contents      

• Quality of Clinical Guidelines »
• Poor Quality of Guidelines: Case Study — The Evidence on Well-Child Care Recommendations »
• newest Another Cautionary Tale About Clinical Guidelines: Screening for Heart Attack Prevention and Education (SHAPE) Task Force Guidelines Example »

Clinical guidelines can improve care and conserve resources, but they vary greatly in quality. Several groups (including Delfini) have created tools to evaluate someone else’s guidelines. A key component of guideline evaluation is the process by which the guideline was developed. In a recent issue of the Journal of Clinical Epidemiology, Giannakakis et al report that although the use of RCTs in developing clinical guidelines has increased, almost half of the guidelines appearing in journals such as the Annals of Internal Medicine, BMJ, JAMA, NEJM, Lancet and Pediatrics do not cite RCTs! Because observational studies and expert opinion are usually insufficient to draw conclusions about effectiveness (cause/effect relationship between intervention and outcomes), any prevention or treatment guidelines not based on RCTs may have major problems with validity.

To read the abstract of this article, go to —

J Clin Epidemiol 2002 Jun;55(6):545-55
Citation of randomized evidence in support of guidelines of therapeutic and preventive interventions.
Giannakakis IA, Haidich AB, Contopoulos-Ioannidis DG, Papanikolaou GN, Baltogianni MS, Ioannidis JP.

http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=
Retrieve&db=PubMed&list_uids=12063096&dopt=Abstract

Poor Quality of Guidelines: Case Study — The Evidence on Well-Child Care Recommendations

An article in the December 2004 issue of PEDIATRICS illustrates a common problem in health care—the gap between expert’s opinions and recommendations and quality scientific medical evidence. Moyer and Butler reviewed recommendations for well-child care made by seven major North American organizations and compared them to the best available evidence (systematic reviews of randomized controlled trials and individual randomized controlled trials).

Results
Forty-two preventive interventions were recommended by two or more of these organizations; however, clinical trials have been conducted for only two of these recommended screening interventions.

Some of the key points made by the authors are --

• There is evidence to support the use of folate to prevent neural tube defects.
• No trials were found supporting the other recommended prophylactic interventions.
• There is some evidence that intensive counseling can change some health risk behaviors -- repeated intensive counseling is most likely to be effective. However, some behavioral counseling interventions were shown to have harmful effects.
• Trial data regarding benefits of iron supplementation for development outcomes is lacking.

Comments
This article is useful in that it points out the ongoing problem of making recommendations which are not based on valid, relevant evidence. The authors also note that implementing unproven interventions can result in harm and wasted resources. They make a plea for including the supporting evidence when making recommendations.

Moyer VA and Butler M. Gaps in the Evidence for Well-Child Care: A Challenge to Our Profession. PEDIATRICS;114 No. 6 : 1511-1521. PMID: 15574609

Another Cautionary Tale About Clinical Guidelines: Screening for Heart Attack Prevention and Education (SHAPE) Task Force Guidelines Example
03/23/08

The Screening for Heart Attack Prevention and Education (SHAPE) Task Force guidelines[1] are clinical recommendations developed through a consensus process by a group of health care professionals and others. The lead author of the SHAPE guidelines is the CEO and President of Endothelix. Endothilix is medical device company “dedicated to human vascular health, with particular emphasis on screening for and monitoring vascular endothelial dysfunction.”[2] There are several unique features of these guidelines. They go far beyond any other cardiovascular risk screening guidelines and “call for” the following:

• Noninvasive screening of all asymptomatic men 45–75 years of age and asymptomatic women 55–75 years of age (except those defined as very low risk) to detect and treat those with subclinical atherosclerosis through…
• A variety of screening tests including measurement of coronary artery calcification by computed tomography scanning and carotid artery intima–media thickness and plaque by ultrasonography.

The authors recommend careful and responsible implementation of these tests as part of a comprehensive risk assessment and reduction approach. They do not, however, provide any evidence of improved outcomes for patients undergoing these tests and state that cost-effectiveness of the use of these tests in a comprehensive strategy must be validated.

In his commentary on the SHAPE Task Force guidelines in the January 9, 2008 issue of the Journal of the American Medical Association (JAMA)[3], Peter Jacobson points out that SHAPE’s recommendations are bad public policy, in part, because they lack evidence for improved outcomes and the possibility of harms outweighing benefits. Jacobson suggests that guidelines be “vetted” by professional societies as a way of establishing validity and clinical usefulness.

At first blush this sounds reasonable. But unfortunately this will not solve the problem of unproven clinical recommendations. We have seen numerous examples of guidelines published by professional societies which are based on weak or fatally flawed evidence such as in the areas of cholesterol screening in children, treatment of Bell’s Palsy and interventions for chronic spinal pain. Others have reported similar findings.[4] Grilli et al.[5], in a review of 431 guidelines produced by U.S. medical societies, found that —

• 87% did not report whether a systematic search of the literature was performed;
• 82% did not apply explicit criteria to grade their evidence;
• 67% did not describe the type of professionals involved in the development of the guideline.
• Acceptance of unproven interventions is partly due to a fairly universal lack of effective training for the health professions in understanding scientific validity.

We recommend that users of guidelines and other clinical recommendations scrutinize all such documents for validity and usefulness rather than relying on endorsement by professional societies, reliance upon which is reasonably likely to lead to outcomes as problematic as Jacobsen suggests.

At a minimum, we suggest the pertinent questions are these:

1. Are the recommendations rigorously evidence-based and their development transparent? This requires understanding principles of scientific validity and should include performing a critical appraisal audit of the science upon which the recommendations are based. There are many resources available to help readers evaluate information sources for validity.
2. Is this information relevant to patients’ needs? Are the expected outcomes clinically significant and will they provide reasonable estimates of benefit? Are the important recommendations/options (with benefits, risks, harms, uncertainties, alternatives and costs of each choice) provided? Do the recommendations accommodate differing patient values and preferences?
3. Can this improvement be implemented and is it likely to succeed? How will the guideline impact outcomes in the setting in which it is applied? Can one measure the effect of implementation?
4. How current is this document?
5. Who developed the recommendations? Were epidemiologic and clinical perspectives included? Were other disciplines and perspectives represented as needed?
6. Are any limitations described?
7. Are there ethical issues to be considered?[6]

Although peer-review may be desirable, we believe that the type of checklist above provides a more appropriate solution for the evaluation of clinical guidelines for validity and clinical usefulness than specialty society endorsement because it helps remove the bias which may be present in any group making clinical recommendations based on consensus and low quality evidence.[7] We also wholeheartedly agree that “guidelines” should not be mandated through legislation. In the case of the SHAPE recommendations, Texas House Representative Rene Oliveira plans to introduce in 2008 his bill into state legislature. The bill would mandate insurers to cover screening of asymptomatic atherosclerosis using calcium scanning and carotid ultrasound, as recommended in the SHAPE initiative. Let’s hope the Texas legislature is awake and alert.

1. Naghavi M, Falk E, Hecht HS, et al. From vulnerable plaque to vulnerable patient, part III: executive summary of the Screening for Heart Attack Prevention and Education (SHAPE) Task Force Report. Am J Cardiol. 2006;98(2A):2H-15H.
2. http://www.endothelix.com/companyprofile.html. Accessed 3/22/08.
3. Jacobson PD. Transforming clinical practice guidelines into legislative mandates. Proceed with abundant caution. JAMA 2008; 299:208-210.
4. Giannakakis IA, Haidich AB, Contopoulos-Ioannidis DG, Papanikolaou GN, Baltogianni MS, Ioannidis JP. Citation of randomized evidence in support of guidelines of therapeutic and preventive interventions. J Clin Epidemiol. 2002 Jun;55(6):545-55.
5. Grilli R, Magrini N, Penna A, Mura G, Liberati A. Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet. 2000 Jan 8;355(9198):103-6.
6. Adapted from Project Appraisal Tool. http://www.delfini.org/delfiniTools.htm. Accessed 1/15/08.
7. Kaptchuk TJ. Effect of interpretive bias on research evidence. BMJ. 2003 Sep 27;327(7417):752.