Evidence Based Health Care Quality Help: EBM Facts

A Few More Details

Most of us know that there is much variation in healthcare that is not explained by patient preference, differences in disease incidence or resource availability. However did you know—

Problems

Fact 1. Healthcare in the United States is shockingly poor.[1-5]

Leading experts estimate that 20 to 50 percent of all healthcare in the United States is inappropriate. This leads to patient harms and waste (at least 100 billion dollars annually). Here are some big reasons why we think there is so much inappropriate care...

Fact 2. Training in medical schools and other schools for allied health professionals in the United States is shockingly poor when it comes to training in science.

Training of healthcare professionals is generally lacking in understanding medical science. As a result, the majority of physicians, clinical pharmacists and others working in health care medical decision-making, (and this includes academicians, researchers, editors and peer reviewers) do not know how to conduct a reliable study nor how to evaluate whether a research study is valid and the results, clinically useful. Most who believe they have these skills do not. Read our Report on Critical Appraisal Skills Failure Rates.

Fact 3. This affects the quality of medical research and the quality of medical care.

We have long estimated that less than 10 percent of all medical research—regardless of source—is reliable or clinical useful. Others agree. Professor John Ioannidis "...charges that as much as 90 percent of the published medical information that doctors rely on is flawed.” We think this is just the tip of the iceberg. Read Delfini Letter and Commentary on "Lies, Damned Lies, and Medical Science," by David H. Freedman, The Atlantic, November 2010. [6, 7] In one survey of 60,352 studies, a meager 7% passed criteria of high quality methods and clinical relevancy. [8] Of 1,165 studies reviewed from 2006 to 2nd quarter of 2010, only 8% were judged reliable by an evidence-savvy insurer.[9]

Fact 4. There is nowhere to turn for trustable healthcare information.

FDA approval is not sufficient for establishing scientific validity and usefulness.

We know of no fully "trustable" healthcare information sources, and sources that claim to be "evidence-based" frequently are not. Some of the best and "most trusted" sources have frequently failed our critical appraisal audits.

Most secondary sources are based on invalid studies or studies that do not have clinically meaningful outcomes. This includes reviews, meta-analyses, performance measures, clinical recommendations, pharmacoeconomic studies, disease management protocols and more. Clinical guidelines vary in quality and the majority may be invalid, including many from professional societies.

Fact 5. Possessing skills in critical appraisal matters.

Bias in studies tends to favor the intervention under investigation. Certain kinds of bias have been shown to distort research results up to a relative 50 percent or more—for each flaw. [10: References for the evidence on the distorting effects of bias available upon request.]

Furthermore, there are a lot of other problems that possessing critical appraisal skills are likely to solve—here are just a few examples...

Most physicians rely on abstracts which are frequently inaccurate. One study found that 18-68 percent of abstracts in 6 top-tier medical journals contained information not verifiable in the body of the article.[11] One study concluded that there may be considerable bias in p-values reported in abstracts.[12] Physicians and others who understand critical appraisal know it cannot be determined whether a study is valid by reading the abstract.

Many physicians rely on information that should be treated as hypothesis-generating only. Physicians who understand critical appraisal know how to avoid being mislead by this.

Author's conclusions are opinions, not evidence—and authors are often biased, even with the best of intentions. Authors frequently use misleading terms or draw misleading conclusions. Physicians and others who lack critical appraisal skills often get "had" by this, whereas critical appraisal knowledge can frequently defend against these problems.

Physicians and others who do not understand issues with findings that are not statistically significant frequently mistakenly interpret these findings as meaning there is no meaningful difference between the groups. Those with critical appraisal skills understand how to use confidence intervals to avoid these erroneous interpretations.

The Solution

Fact 6. Attaining critical appraisal skills need not be hard or time-consuming—individuals can easily acquire the basic skills and without learning much about statistics.

Fact 7. Skills can be easily acquired in a short period of time.

Healthcare providers are owe it to their patients to gain these skills. There are many resources available—both formal and informal—including self-teaching modules available online. We provide a wealth of materials online for self-study for free.

We also offer fee-based services and can teach people basic skills for analyzing studies of interventions in a short period of time.

Our programs are not boring, nor are they too hard. We utilize a practical, simplified and applied tool-based approach in an engaging and entertaining way.

"...thanks for presenting this valuable information in a fun and memorable way..."

"...I thought the subject was going to be boring and not much use...I was very wrong...I cannot thank you and Sheri enough."

More Testimonials.

REFERENCES

1. Chassin MR, Galvin RW, the National Roundtable on Health Care Quality. The urgent need to improve health care quality: Institute of Medicine National Roundtable on Health Care Quality [consensus statement]. JAMA. 1998;280:1000–1005.

2. Skinner J, Fisher ES, Wennberg JE; for the National Bureau of Economic Research. The efficiency of Medicare. Working Paper No. 8395. Cambridge, MA: National Bureau of Economic Research; July 2001.

3. McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348:2635–2645.

4. Kerr EA, McGlynn EA, Adams J, et al. Profiling the quality of care in twelve communities: Results from the CQI study. Health Aff. 2004;23:247–256.

5. Centers for Medicare and Medicaid Services, Office of the Actuary, National Health Statistics Group, National Health Care Expenditures Data, January 2010.

6. Strite SA, Stuart ME. Letter to the Editor: How to Evaluate Medical Science. The Atlantic. January/February 2011.

7. Freedman DH. Lies, Damned Lies, and Medical Science. The Atlantic. November 2010.

8. McKibbon KA, Wilczynski NL, Haynes RB. What do evidence-based secondary journals tell us about the publication of clinically important articles in primary healthcare journals? BMC Med. 2004 Sep 6;2:33. PubMed PMID: 15350200; PubMed Central PMCID: PMC518974.

9. RegenceRx. Presented at the RegenceRx.Delfini Evidence-based Collaborative. Bridgewater NJ. March 16, 2011

10. References for the distorting effects of bias are available upon request to delfini (at) delfini.org.

11. Pitkin RM, Branagan MA, Burmeister LF. Accuracy of Data in Abstracts of Published Research Articles. JAMA. 1999; 281: 1110-1111. [PMID 10188662]

12. Gøtzsche PC. Believability of relative risks and odds ratios in abstracts: cross sectional study. BMJ 2006;333;231-234; PMID: 16854948

Fact Sheet Quality of Care, Quality of Science & Importance of Critical Appraisal Skills
Get Our Updates! Key Terms Critical appraisal of the medical literature means evaluating a research study for validity (meaning its likelihood of being true) and clinical usefulness (meaningful benefit to patients). Performing critical appraisal combines understanding of critical appraisal concepts + clinical understanding + critical thinking. Validity means probably true. Outcomes are what we are interested in studying (such as reduction in mortality or increase in happiness!) Clinical usefulness means benefiting a patient in meaningful ways (size of the results) in the areas of morbidity, mortality, symptom relieve, emotional or physical functioning, or health-related quality of life. Primary studies are the reports of original research. Secondary studies are studies of primary studies. A meta-analysis is an example of a secondary study as is a systematic review. Secondary sources are any information sources that utilize primary or secondary studies. Bias means that which leads away from truth that isn't a random chance accident. Chance means just that: that the results are just a random event. Quality Science Matters [References available upon request] Roughly 58,000 US lives were lost in the Vietnam war. Delfini estimates that possibly ~63,000 preventable deaths were due to encainide/fleccainide for premature ventricular contractions (PVCs) after acute myocardial infarction. Others estimate that Vioxx “may have contributed to 27,785 heart attacks and sudden cardiac deaths between 1999 and 2003.” Roughly 42,000 women with advanced breast cancer were subjected to treatment with autologous bone marrow transplant and high dose chemotherapy. It is estimated that over 9,000 died from treatment. Yet, RCTs showed no benefit. Costs have been estimated at $3.4 billion. Understanding basic critical appraisal methods may have prevented these deaths which were potentially attributable to these agents.	The Short Story About the Huge Problem We are Trying to Help Solve Delfini Flier A Few More Details Most of us know that there is much variation in healthcare that is not explained by patient preference, differences in disease incidence or resource availability. However did you know— Problems Fact 1. Healthcare in the United States is shockingly poor.[1-5] Leading experts estimate that 20 to 50 percent of all healthcare in the United States is inappropriate. This leads to patient harms and waste (at least 100 billion dollars annually). Here are some big reasons why we think there is so much inappropriate care... Fact 2. Training in medical schools and other schools for allied health professionals in the United States is shockingly poor when it comes to training in science. Training of healthcare professionals is generally lacking in understanding medical science. As a result, the majority of physicians, clinical pharmacists and others working in health care medical decision-making, (and this includes academicians, researchers, editors and peer reviewers) do not know how to conduct a reliable study nor how to evaluate whether a research study is valid and the results, clinically useful. Most who believe they have these skills do not. Read our Report on Critical Appraisal Skills Failure Rates. Fact 3. This affects the quality of medical research and the quality of medical care. We have long estimated that less than 10 percent of all medical research—regardless of source—is reliable or clinical useful. Others agree. Professor John Ioannidis "...charges that as much as 90 percent of the published medical information that doctors rely on is flawed.” We think this is just the tip of the iceberg. Read Delfini Letter and Commentary on "Lies, Damned Lies, and Medical Science," by David H. Freedman, The Atlantic, November 2010. [6, 7] In one survey of 60,352 studies, a meager 7% passed criteria of high quality methods and clinical relevancy. [8] Of 1,165 studies reviewed from 2006 to 2nd quarter of 2010, only 8% were judged reliable by an evidence-savvy insurer.[9] Fact 4. There is nowhere to turn for trustable healthcare information. FDA approval is not sufficient for establishing scientific validity and usefulness. We know of no fully "trustable" healthcare information sources, and sources that claim to be "evidence-based" frequently are not. Some of the best and "most trusted" sources have frequently failed our critical appraisal audits. Most secondary sources are based on invalid studies or studies that do not have clinically meaningful outcomes. This includes reviews, meta-analyses, performance measures, clinical recommendations, pharmacoeconomic studies, disease management protocols and more. Clinical guidelines vary in quality and the majority may be invalid, including many from professional societies. Fact 5. Possessing skills in critical appraisal matters. Bias in studies tends to favor the intervention under investigation. Certain kinds of bias have been shown to distort research results up to a relative 50 percent or more—for each flaw. [10: References for the evidence on the distorting effects of bias available upon request.] Furthermore, there are a lot of other problems that possessing critical appraisal skills are likely to solve—here are just a few examples... Most physicians rely on abstracts which are frequently inaccurate. One study found that 18-68 percent of abstracts in 6 top-tier medical journals contained information not verifiable in the body of the article.[11] One study concluded that there may be considerable bias in p-values reported in abstracts.[12] Physicians and others who understand critical appraisal know it cannot be determined whether a study is valid by reading the abstract. Many physicians rely on information that should be treated as hypothesis-generating only. Physicians who understand critical appraisal know how to avoid being mislead by this. Author's conclusions are opinions, not evidence—and authors are often biased, even with the best of intentions. Authors frequently use misleading terms or draw misleading conclusions. Physicians and others who lack critical appraisal skills often get "had" by this, whereas critical appraisal knowledge can frequently defend against these problems. Physicians and others who do not understand issues with findings that are not statistically significant frequently mistakenly interpret these findings as meaning there is no meaningful difference between the groups. Those with critical appraisal skills understand how to use confidence intervals to avoid these erroneous interpretations. The Solution Fact 6. Attaining critical appraisal skills need not be hard or time-consuming—individuals can easily acquire the basic skills and without learning much about statistics. Fact 7. Skills can be easily acquired in a short period of time. Healthcare providers are owe it to their patients to gain these skills. There are many resources available—both formal and informal—including self-teaching modules available online. We provide a wealth of materials online for self-study for free. We also offer fee-based services and can teach people basic skills for analyzing studies of interventions in a short period of time. Our programs are not boring, nor are they too hard. We utilize a practical, simplified and applied tool-based approach in an engaging and entertaining way. "...thanks for presenting this valuable information in a fun and memorable way..." "...I thought the subject was going to be boring and not much use...I was very wrong...I cannot thank you and Sheri enough." More Testimonials. REFERENCES 1. Chassin MR, Galvin RW, the National Roundtable on Health Care Quality. The urgent need to improve health care quality: Institute of Medicine National Roundtable on Health Care Quality [consensus statement]. JAMA. 1998;280:1000–1005. 2. Skinner J, Fisher ES, Wennberg JE; for the National Bureau of Economic Research. The efficiency of Medicare. Working Paper No. 8395. Cambridge, MA: National Bureau of Economic Research; July 2001. 3. McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348:2635–2645. 4. Kerr EA, McGlynn EA, Adams J, et al. Profiling the quality of care in twelve communities: Results from the CQI study. Health Aff. 2004;23:247–256. 5. Centers for Medicare and Medicaid Services, Office of the Actuary, National Health Statistics Group, National Health Care Expenditures Data, January 2010. 6. Strite SA, Stuart ME. Letter to the Editor: How to Evaluate Medical Science. The Atlantic. January/February 2011. 7. Freedman DH. Lies, Damned Lies, and Medical Science. The Atlantic. November 2010. 8. McKibbon KA, Wilczynski NL, Haynes RB. What do evidence-based secondary journals tell us about the publication of clinically important articles in primary healthcare journals? BMC Med. 2004 Sep 6;2:33. PubMed PMID: 15350200; PubMed Central PMCID: PMC518974. 9. RegenceRx. Presented at the RegenceRx.Delfini Evidence-based Collaborative. Bridgewater NJ. March 16, 2011 10. References for the distorting effects of bias are available upon request to delfini (at) delfini.org. 11. Pitkin RM, Branagan MA, Burmeister LF. Accuracy of Data in Abstracts of Published Research Articles. JAMA. 1999; 281: 1110-1111. [PMID 10188662] 12. Gøtzsche PC. Believability of relative risks and odds ratios in abstracts: cross sectional study. BMJ 2006;333;231-234; PMID: 16854948	Menu in Brief ........ Use of our website implies agreement to our Notices. Delfini Home Critical Appraisal Skills Failure Rates Delfini Seminars 2011 Click on Our Delfini Flier Critical Appraisal Tutorial ........ An evidence-based approach can improve patient care and reduce waste. It is the right thing to do. There can be a return-on-investment (ROI) for EBM. ........ Read our Report on Critical Appraisal Skills Failure Rates Free Critical Appraisal Help: Resources What are the core competencies required for critical appraisal for studies of interventions? Download our 1-page critical appraisal checklist and find out: [WORD] & [PDF] Delfini tools can be found at the Delfini Library. ........ Home What's New Services Resources Success Stories Notices About Us About Our Work Testimonials Other Site Navigator Contact Us *Site Search* ........ Quick Navigator to Selected Resources: Tools & Education Tutorials Favorites Evidence Essentials *DelfiniClick™ On the Same* Page™ Messaging Scripts *Delfini* Publications Glossary Reading Links Delfini Flier Get DelfiniGram Updates
Delfini SERVICES We offer a wide range of supportive services with the ultimate goal of increasing health care organizations’ and clinicians’ abilities to provide high quality, evidence-based care and to improve medical decision-making for organizations, leaders, teams, clinicians and patients. Delfini About Our Work Who We Can Help Our Capabilities Services Success Stories Testimonials About Us Contact Us Resources at — delfini.org Resources medicalleaders.org
Return to Top © 2002-2012 Delfini Group, LLC. All Rights Reserved Worldwide. Dolphin photo for Dolphins Poolside attributed to RevolverOcelo. Delfini Boat thanks to Dr. Bat Shunatona. Thanks too to contributors to Openclipart.org and Wiki Media. Use of this website implies your agreement to our Notices. EBM Solutions for Evidence Based Health Care Quality