Delfini Fact Sheet: Quality of Care, Quality of Science & Importance of Critical Appraisal Skills

• It is believed that most medical interventions lack scientific evidence of efficacy.
• The majority of medical research studies fail validity criteria — even those published in best journals or best sources.
• FDA approval is not sufficient for establishing scientific validity and usefulness.
• Most physicians rely on abstracts which are frequently inaccurate and from which validity cannot be assessed.
  • One study found that 18-68% of abstracts in 6 top-tier medical journals contained information not verifiable in body of the article..
    o One study concluded that there may be considerable bias in p-value reported in abstracts.
• Many physicians rely upon information that should only be used for hypothesis generation
  • Narrative reviews and overviews cannot be relied upon for providing valid evidence
  • Case reports, case series or observational studies should only be considered hypothesis-generating, but are frequently used by clinicians to draw conclusions.
  • Observational studies are especially prone to bias and are highly likely to mislead when used for interventions.
• Bias tends to inflate study results by up to a relative 15 to 50 percent — examples:
  • Lack of blinding, alone, is likely to overestimate benefit by a relative 17% to 44%.
  • Lack of randomization was shown in one study to falsely inflate outcomes from 0% to a 10% absolute difference.
  • One researcher reports potentially erroneous misrepresentation of outcomes by a relative difference of 50% or higher.

• Nearly ALL of published medical science — regardless of source or support — is not valid (and by "valid", meaning "probably true") and is therefore misleading, is insufficiently reported to determine if it is valid or is not clinically useful - meaning the results are not likely to help patients.
• This applies not only to research studies, but also to meta-analyses and health information sources such as guidelines even from professional medical societies: "evidence-based" is too often misapplied either through lack of knowledge or because it is being used as a buzz word.
• Critical appraisal of medical science matters because studies that are not valid are likely to overestimate benefit making interventions appear more effective than they are or effective when they are not at all: the failure to avoid many key biases has been estimated to potentially inflate results by up to a relative 50 percent for each studied bias.
• Nearly ALL physicians, clinical pharmacists and others working in health care medical decision-making, (and we believe this includes academicians, researchers, editors and peer reviewers) do not know how to evaluate whether a research study is valid and how to assess if the results are clinically useful.
• Most people who think they know how to evaluate the validity of a research study do not.
• Critically appraising studies for validity and clinical usefulness makes a significant difference in patient care.
• It is okay to make health care decisions for reasons other than the scientific evidence — but it is important to know the available evidence first — and to not misstate other reasons as “evidence.”

• An evidence-based approach & critical appraisal skills can improve patient care and reduce waste.
• Attaining critical appraisal skills need not be hard or time-consuming — individuals can easily acquire the basic skills and without learning much about statistics.
• Organizations can successfully apply an evidence- and value-based approach to quality of care.
• If you are a medical leader or a health care provider, you owe it to your patients to learn these skills and apply them. We provide free online help and offer consulting services.
• We have the evidence-on-the-evidence. Contact Sheri for more details at sstrite (at) delfini.org.
  

• Leading experts estimate that 20 to 50 percent of all health care in the US is inappropriate.
• It is reasonable to conclude that much of this inappropriate care can be attributed to faulty clinical judgments and prescribing decisions.
• It is also reasonable to conclude that the quality of patient care can be improved significantly by the application of valid (meaning "probably true") and clinically useful scientific information.
• Unfortunately, the quality of medical scientific information is extremely poor.

• Leading experts estimate that less than 10 percent of all published medical science is valid and clinically useful — even in the best medical journals.
• Most researchers lack sufficient understanding of validity to design and execute valid studies, and they frequently lack knowledge of the importance of reporting key details to enable users to determine validity.
• The problems in quality of science are wide spread and are not limited to any specific group. Everyone has a bias including industry and academia.
• Unfortunately, there is a general lack of awareness of this enormous problem — and there is an unfortunately lack of skills to deal with it. This is why we formed Delfini.

• Surprisingly, many - and possibly most - publishers, editors and peer reviewers lack skills to determine validity and clinical usefulness.
• Also surprisingly, most health care professionals - including clinicians and clinical pharmacists - lack even basic skills to determine validity and clinical usefulness.
• Roughly 75% of physicians and other health care professionals fail our simple three-question critical appraisal quiz.
• And roughly 76% who answered “Yes, I feel confident in my ability to evaluate the medical literature,” missed 2 or 3 out of the 3 basic questions
• See our PDF: Report on EBM Quiz Failures
• Unfortunately, there is currently no place to rely upon for dependably valid and clinically useful information.

• Sources that claim to be "evidence-based" frequently are not.
• There no are medical information sources that should be relied upon without auditing the studies utilized to generate conclusions. Even the best and "most trusted" sources have frequently failed our audits.
• Most secondary sources that utilize primary science are based on invalid studies or studies which do not have clinically meaningful outcomes.
• Clinical guidelines vary in quality and the majority may be invalid, including those from professional societies which, frequently, are not based on valid evidence.
• This includes reviews, meta-analyses, performance measures, clinical recommendations, pharmacoeconomic studies, pathways, disease management protocols, and more. Any source that utilizes medical science...

• Bias tends to favor the intervention, making interventions appear effective when they are not or to be more effective than they actually are.
• Results of studies have supported the conclusion that bias can inflate benefits by up to a relative 40 or 50 percent.
• These high rates of falsely inflated results have been found when study methods such as generation of the randomization sequence, concealment of allocation, blinding, and assessing outcomes through statistical modeling are omitted or not done correctly.
• It is not enough to utilize studies which state they are randomized, double-blinded controlled trials — details of characteristics such as randomization and blinding must be disclosed to be evaluable by users.

• Authors’ conclusions may frequently mislead.
• Did you know that non-statistically significant differences in studies of harms are frequently inappropriately interpreted as no difference?
• Did you know that authors often use terminology that cannot be trusted?
• Author's conclusions are opinions, not evidence — and authors are often biased, even with the best of intentions.

• An evidence-based approach can improve patient care and reduce waste, and there is a return-on-investment (ROI) for EBM

• There are many resources available — both formal and informal — including self-teaching modules available online.
• We can teach people basic skills for analyzing studies of interventions in a half-day.

• Organizations that create evidence- and value-based structures can improve patient care and reduce waste — examples:
• Pharmacy & Therapeutics committees can help reduce waste and potentially improve patient care.
• An informal survey has shown that having an operating Pharmacy & Therapeutics Committee can reduce drug expense. Findings: 10% of premiums go to Rx drug costs in a well-run health care organization with a P & T committee and formulary system; 25% go to Rx drug costs without a P & T committee.
• Read our Messages for P & T Committees
• Download our Evidence-based Pharmacotherapy Book Chapter: "Applying Evidence-based Pharmacotherapy to Formulary Decisions" or our short paper in Formulary on "Process steps and suggestions for creating drug monographs and drug class reviews in an evidence-based formulary system"
• Medical Technology Assessment committees can help reduce waste and potentially improve patient care.
• Establishing an operating Medical Technology Assessment Committee which relies on valid and clinically useful science can improve patient care and reduce the use of unproven technologies along with their associated expenses. Our experience: We estimate that the approval rate for new technologies will drop by an absolute 10 to 15% in organizations that develop an evidence-based evaluation based on our experience at Group Health.
• Read our Messages for Medical Technology Assessment Committees
• Examples of organizations successfully applying an evidence-based approach:
• Guideline: Kaiser Permanente Hawaii Evidence-based Medicine Nephrology Chronic Kidney Disease Project: Project Improvement Initiative and Clinical Practice Guideline
• Regence Group
• McKesson (Interqual)

• See our online Resources
• Including —
• See our Health Care Quality System Assessments help
• "The Evidence-based Organization" published in the Academy of Physician Executive's, The Physician Executive, Journal of Medical Management
• See our Services
• Including —
• Medical Leaders & Other Health Care Leaders and Professionals
• Creating the EBM Organization

Return to Top

© Delfini Group, LLC, 2002-2010. All Rights Reserved Worldwide.
Use of this website implies your agreement to our Notices.

EBM Solutions for Evidence Based Health Care Quality