Critical Appraisal Matters
The EBM Information Quest: Is it true? Is it useful? Is it usable?™
Follow & Share...
At "Why Critical Appraisal Matters"
Quick Navigator to Selected Resources
Key Reading: "How To Survive the Medical Misinformation Mess" by John P. A. Ioannidis, Michael E. Stuart, Shannon Brownlee, Sheri A. Strite, published in the European Journal of Clinical Investigation and, within fewer than four months after publication, one of the top 20 papers in 2017 downloaded from The European Journal of Clinical Investigation.
Much medical science is not reliable or of uncertain reliability. Worse, health care providers frequently do not know how to detect unreliable medical science. Health care providers frequently do not know how to interpret study results—even basic ones. Roughly, over 70 percent of physicians, clinical pharmacists and other health care professionals fail our simple 3-question critical appraisal training program pre-test [Delfini].
To give you a flavor of how serious this failure is, one question simply assesses the person's awareness of a need for a comparison group in an efficacy study. Without a comparison group, how would you know a condition wasn't self-limiting, for example, and recovery not due to an intervention?
Understanding basic requirements for valid medical science is very important. Health care providers owe it to patients to acquire these skills. Otherwise, patients can be harmed or miss opportunities for true consent.
Numbers below are estimates, but are likely to be close or understated. To put these numbers into context, roughly 58,000 US lives were lost in the Vietnam war. [ref last accessed 10/08/2012]
In the above instances, these were preventable deaths and harms—from common treatments—which patients might have avoided if their physicians had better understood the importance and methods of evaluating medical science.
Most of us know that there is much variation in healthcare that is not explained by patient preference, differences in disease incidence or resource availability. However did you know—
Fact 1. Healthcare in the United States is shockingly poor. [Chassin, CMS 10, Kerr, McGlynn, Skinner]
Leading experts estimate that 20 to 50 percent of all healthcare in the United States is inappropriate. This leads to patient harms and waste (at least 100 billion dollars annually). Here are some big reasons why we think there is so much inappropriate care...
Fact 2. Training in medical schools and other schools for allied health professionals in the United States is shockingly poor when it comes to training in science.
Training of healthcare professionals is generally lacking in understanding medical science. Douglas Altman wrote of this problem many years ago in an editorial titled "The scandal of poor medical research" [Altman] which was recently remarked on by Richard Smith in a BMJ blog entited, "Medical research—still a scandal." [Smith]
As a result, the majority of physicians, clinical pharmacists and others working in health care medical decision-making, (and this includes academicians, researchers, editors and peer reviewers) do not know how to conduct a reliable study nor how to evaluate whether a research study is valid and the results, clinically useful. Most who believe they have these skills do not. Read our Report on Critical Appraisal Skills Failure Rates.
Fact 3. This affects the quality of medical research and the quality of medical care.
We have long estimated that less than 10 percent of all medical research—regardless of source—is reliable or clinical useful. Others agree. Professor John Ioannidis "...charges that as much as 90 percent of the published medical information that doctors rely on is flawed.” We think this is just the tip of the iceberg. Read Delfini Letter and Commentary on "Lies, Damned Lies, and Medical Science," by David H. Freedman, The Atlantic, November 2010. [Strite, Freedman] In one survey of 60,352 studies, a meager 7% passed criteria of high quality methods and clinical relevancy [McKibbon], and fewer than 5% passed a validity screening for an evidence-based journal. [Glasziou]
Fact 4. There is nowhere to turn for trustable healthcare information.
FDA approval is not sufficient for establishing scientific validity and usefulness.
We know of no fully "trustable" healthcare information sources, and sources that claim to be "evidence-based" frequently are not. Some of the best and "most trusted" sources have frequently failed our critical appraisal audits.
Most secondary sources are based on invalid studies or studies that do not have clinically meaningful outcomes. This includes reviews, meta-analyses, performance measures, clinical recommendations, pharmacoeconomic studies, disease management protocols and more. Clinical guidelines vary in quality and the majority may be invalid, including many from professional societies.
Fact 5. Possessing skills in critical appraisal matters.
Bias in studies tends to favor the intervention under investigation. Certain kinds of bias have been shown to distort research results up to a relative 50 percent or more—for each flaw. [References for the evidence on the distorting effects of are available upon request to delfini (at) delfini.org.]
Furthermore, there are a lot of other problems that possessing critical appraisal skills are likely to solve—here are just a few examples...
Most physicians rely on abstracts which are frequently inaccurate. One study found that 18-68 percent of abstracts in 6 top-tier medical journals contained information not verifiable in the body of the article.[Pitkin] One study concluded that there may be considerable bias in p-values reported in abstracts.[Gøtzsche] Physicians and others who understand critical appraisal know it cannot be determined whether a study is valid by reading the abstract.
Many physicians rely on information that should be treated as hypothesis-generating only. Physicians who understand critical appraisal know how to avoid being mislead by this.
Author's conclusions are opinions, not evidence—and authors are often biased, even with the best of intentions. Authors frequently use misleading terms or draw misleading conclusions. Physicians and others who lack critical appraisal skills often get "had" by this, whereas critical appraisal knowledge can frequently defend against these problems.
Physicians and others who do not understand issues with findings that are not statistically significant frequently mistakenly interpret these findings as meaning there is no meaningful difference between the groups. Those with critical appraisal skills understand how to use confidence intervals to avoid these erroneous interpretations.
Attaining critical appraisal skills need not be hard or time-consuming—individuals can easily acquire the basic skills and without learning much about statistics. Skills can be easily acquired in a short period of time.
Healthcare providers owe it to their patients to gain these skills. There are many resources available—both formal and informal—including self-teaching modules available online. We provide a wealth of materials online for self-study for free and have several books available for purchase.
We also offer fee-based services and can teach people basic skills for analyzing studies of interventions in a short period of time.
Our programs are not boring, nor are they too hard. We utilize a practical, simplified and applied tool-based approach in an engaging and entertaining way.
“...thanks for presenting this valuable information in a fun and memorable way...”
"...I thought the subject was going to be boring and not much use...I was very wrong...I cannot thank you and Sheri enough."
Here are some resources for new learners:
1-Pager: Basics of Evaluating Evidence in Superiority Trials for Therapies [PDF]
Short critical appraisal tutorial: EVALUATING BIASED EFFICACY RESULTS IN SUPERIORITY TRIALS
We also have an online tutorial and videos on critical appraisal here.