Contents      

About PMID Numbers: We frequently utilize a PMID number in place of a citation. Where PMID numbers are available, enter that number into the PubMed search box to retrieve that citation and listing.

Special Feature: The Validity Detective

Quality of Evidence

Observational Studies
More on the Problem with Drawing Cause-Effect Conclusions from Observational Studies »
Cause & Effect Conclusions from Observations »
Bias in Observational Studies — More on HRT in Menopause »

Primary Studies & General Concepts
Understanding Number Needed to Treat (NNT) »
Concealment of Allocation »
Blinding and RCTs »
Blinding in Surgery Trials »
The Importance of Blinded Assessors in RCTs »
Attrition Bias: Intention-to-Treat Basics »
Intention-to-Treat Analysis: Censoring »
Intention-to-Treat Analysis: Misreporting and Migraine »
Missing Data Points: Difference or No Difference »
Quality of Studies: Lower Quality = Greater Effect Size »
   More: Overestimation of Effect Size in Studies of Low Quality »
External Validity: Case of the Carotid Stent »
Quality of Studies: VIGOR »
Confidence-Intervals, Power & Meaningful Clinical Benefit »

Diagnostic Studies
Bias in Diagnostic Studies »

Secondary Studies
Systematic Reviews: Quality & Searching Tips »
Systematic Reviews: Untrustable "Trustable" Sources? »
Delfini Letter to BMJ: Corticosteriods are Not Proven for Treatment of Bell's Palsy »
Delfini Letter to NEJM: Bell's Palsy - REDUX! »
Substandard Evidence: POEMS and Diabetes — Readers, Beware! »
Quality of Systematic Reviews: Misleading “POEM” on Hormone Therapy »

Secondary Sources
Quality of Clinical Guidelines »
Poor Quality of Guidelines: Case Study — The Evidence on Well-Child Care Recommendations »

Quality Improvement
Successful Evidence-based QI Project: Diabetes Management at Dreyer Medical Clinical »

For Clicks that do not work, please email us to let us know. For many of them, you may be able to go to PubMed and do a quick search using the citation: www.PubMed.gov.

Confidence-Intervals, Power & Meaningful Clinical Benefit:
Advice to Readers on How to Stop Worrying about Power and Start Using Confidence Intervals &
Using Confidence Intervals to Evaluate Clinical Benefit of Statistically Significant Findings
(Special thanks to Brian Alper, MD, MSPH and Ted Ganiats, MD for their help in understanding this issue.)

This is how confidence intervals are reported:

Example: ARR = 5%; 95% CI (3% to 7%)

Absolute Risk Reduction and Relative Risk Reduction
For measures reported as percentages, if the range includes zero, the outcomes are not statistically significant.

Relative Risk (aka Risk Ratio) and Odds Ratio
For measures reported as ratios, if the range includes 1, the outcomes are not statistically significant.

Example: Clinical Significance Goal
>=1% absolute reduction in mortality

For Non-Significant Findings:

Example 1

• ARR = 2%; 95% CI (-1% to 5%)
• The upper boundary tells you it is possible that the true result WOULD meet your requirements for clinical significance – thus, from that perspective this trial is inconclusive about NO DIFFERENCE BETWEEN GROUPS - you do not know if the trial was insufficiently powered (false negative due to insufficient number of people to show a statistically significant difference if there is one)

Example 2

• ARR = 0%; 95% CI (-.5 to .5%)
• The upper boundary does not reach your goal – therefore, this can be considered sufficient evidence that there is no difference between the groups that you would consider clinically significant

How to Use Confidence Intervals to Determine Conclusiveness of Non-significant Findings
Again, you can also use confidence intervals to determine whether a result from a valid study is of meaningful clinical benefit.

Requirements for Meaningful Clinical Benefit
Remember that outcomes of clinical significance are those which benefit patients in some way in the areas of morbidity, mortality, symptom relief, physical or emotional functioning or health-related quality of life. Intermediate markers are assumed to benefit patients in these areas, but they may not - thus, a direct causal chain of benefit must be proved to avoid waste and potential patient harms occurring as unintended consequences. Meaningful clinical benefit is a combination of benefits in a clinically significant area along with the size of the results.

As with evaluating the conclusiveness of a non-significant finding, you apply judgment to set your minimum requirement for meaningful clinical significance. Using the same example of your choosing 1 percent absolute reduction in mortality as meaningful clinical benefit:

Example: Clinical Significance Goal
>=1% absolute reduction in mortality

For Statistically Significant Findings:

Example 1

• ARR = 2%; 95% CI (.5% to 3.5%)
• The lower boundary tells you it is possible that the true result will NOT meet your requirements for clinical significance – thus, from that perspective this trial is inconclusive

Example 2

• ARR = 2%; 95% CI (1 to 3%)
• The lower boundary reaches your goals for clinical significance – therefore, this can be considered sufficient evidence of benefit

Overestimation of Effect Size in Studies of Low Quality

In a previous DelfiniClick, we summarized an article by Moher and colleagues (1) in which the authors randomly selected 11 meta-analyses involving 127 RCTs which evaluated the efficacy of interventions used for circulatory and digestive diseases, mental health, pregnancy and childbirth. Moher and colleagues concluded that -

• Low-quality trials compared with high quality trials (score >2), were associated with a relative increased estimate of benefit (34%).
• Trials that used inadequate allocation concealment, compared with those that used adequate methods, were associated with a relative increased estimate of benefit (37%).

Below we summarize another study that confirms and expands Moher’s findings. In a study similar to Moher’s, Kjaergard and colleagues (2) evaluated the effects of methodologic quality on estimated intervention effects in randomized trials.

The study evaluated 23 large and 167 small randomized trials and a total of 136,164 participants. Methodologic quality was defined as the confidence that the trial’s design, conduct, analysis, and presentation minimized or avoided biases in the trial’s intervention comparisons (3). The reported methodologic quality was assessed using four separate components and a composite quality scale.

The quality score was ranked as low (</=2points) or high (>/=3 points), as suggested by Moher et al. (1). The four components were 1) generation of allocation sequence; 2) concealment of allocation; 3) double-blinding; and, 4) reporting of loss-to-follow-up:

RESULTS OF KJAERGARD ET AL’S REVIEW (all reported exaggerations are relative increases):

Generation of Allocation Sequence
The odds ratios generated by all trials (large and small) with inadequate generation of the allocation sequence were on average significantly exaggerated by 51% compared with all trials reporting adequate generation of allocation sequence (ratio of odds ratios (95% CI) = 0.49 (0.30–0.81), P <0.001.

Concealment of Allocation
All trials with inadequate allocation concealment exaggerated intervention benefits by 40% compared with all trials reporting adequate allocation concealment (ratio of odds ratios (95% CI) = 0.60 (0.31–1.15), P =0.12. Odds ratios were significantly exaggerated by 52% in small trials with inadequate versus adequate allocation concealment (ratio of odds ratios (95% CI) 0.48 (0.25–0.92), P = 0.027).

Double Blinding
The odds ratios generated by all trials without double blinding were significantly exaggerated by 44% compared with all double-blind trials (ratio of odds ratios (95% CI) = 0.56 (0.33–0.98), P = 0.041).

Reporting of Loss-to-Followup
The analyses showed no significant association between reported follow-up and estimated intervention effects (ratio of odds ratios (95% CI) = 1.50 (0.80–2.78), P = 0.2).

Kjaergard and Colleagues’ Conclusions

1. Adequate generation of the allocation sequence and adequate allocation concealment should be required for adequate randomization.

   Unlike previous investigators (1,3,4, 5), the authors found that trials with inadequate generation of allocation sequence exaggerate intervention effects significantly.

2. Trials with inadequate allocation concealment also generate exaggerated results.

   This is in accordance with previous evidence (1,3,5). The authors found that despite the considerable overlap between generation of allocation sequence and allocation concealment, both factors may independently affect the estimated intervention effect.

3. Trials without double blinding exaggerate results.

   This study supports Schulz and colleagues’ finding of a significant association between intervention effects and double blinding and extends the evidence by including trials from several therapeutic areas.

4. There was no association between reported follow-up and intervention effect.

Delfini Comment
It is useful to know quantitatively how various threats to validity affect results when doing critical appraisal of a study. The study by Kjaergard and colleagues summarized above expands the findings of Schulz, Moher, Juni and others.

Previous studies have questioned the reliability of reported losses to follow-up (5, 6). In accordance with Schulz and colleagues’ results (5), the authors found no association between intervention effects and reported follow-up.
Delfini Note: We have found that losses to follow-up may significantly affect P values when sensitivity analysis is done. We consider loss of =/>5% with differential loss or =/> 10% without differential loss to be an important threat to validity.

In agreement with the findings of Moher and associates (1,3) and Juni and colleagues (7), the authors found that trials with a low quality score on the scale developed by Jadad and colleagues (8) significantly exaggerate intervention benefits.

Kjaergard and colleagues conclude that assessment of methodologic quality should focus on generation of allocation sequence, allocation concealment, and double blinding. Delfini feels this is not sufficient – but appreciates this study as one that further demonstrates the importance of effective approaches to some of these methodologic areas.

References
1. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet. 1998;352:609-13. [PMID: 9746022]

2. Kjaergard LL, John Villumsen J, Gluud C. Reported Methodologic Quality and Discrepancies between Large and Small Randomized Trials in Meta-Analyses. Ann Intern Med. 2001;135:982-989.

3. Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, Jones A, et al. Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. Health Technol Assess. 1999;3:i-iv, 1-98. [PMID: 10374081]

4. Emerson JD, Burdick E, Hoaglin DC, Mosteller F, Chalmers TC. An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. Control Clin Trials. 1990;11:339-52. [PMID: 1963128]

5. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273:408-12. [PMID: 7823387]

6. Gøtzsche PC. Methodology and overt and hidden bias in reports of 196 double-blind trials of nonsteroidal antiinflammatory drugs in rheumatoid arthritis. Control Clin Trials. 1989;10:31-56. [PMID: 2702836]

7. Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282:1054-60. [PMID: 10493204]

8. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17:1-12. [PMID: 8721797]

Missing Data Points: Difference or No Difference — Does it Matter?

We continue to study the "evidence on the evidence" — meaning we are continually on the look out for information which may shed light on the impact on reported outcomes of certain kinds of bias, for example, or information that provides help in how to handle different biases. Missing data points is an issue affecting the majority of studies, but currently there is not clarity on how big an issue this is, especially when there is not a differential loss between groups.

We spoke recently about this issue with John M. Lachin, Sc.D., Professor of Biostatistics and Epidemiology, and of Statistics, The George Washington University, and author. (And then we did some "hard thinking" as David Eddy would say.) Even without differential loss between the groups overall, a differential loss could occur in prognostic variables — and readers are rarely going to have access to data about changes in prognostic characteristics post-baseline reporting. So we continue to offer our conservative approach that loss of around five percent with differential loss is a bias as well as loss of around ten percent or more without differential loss.

For those who are tough and hardy and really want to mull on this, here's our updated white paper on "missingness" [Word] or [PDF]. We welcome further thoughts (or evidence) on this area.

Results

1. Sixteen of the thirty-two randomized controlled trials did not report blinding of outcome assessors when blinding would have been possible.
2. Among the studies with continuous outcome measures, unblinded outcomes assessment was associated with significantly larger treatment effects than blinded outcomes assessment (standardized mean difference, 0.76 compared with 0.25; p = 0.01).
3. In the studies with dichotomous outcomes, unblinded outcomes assessments were associated with significantly greater treatment effects than blinded outcomes assessments (odds ratio, 0.13 compared with 0.42; p < 0.001).
4. This translates into a relative risk reduction of 38% for blinded outcome assessments compared with 71% for unblinded outcome assessments (a difference of 33%).

Conclusion
Unblinded outcomes assessment dramatically inflates the reported benefit of effectiveness of treatments.

Delfini Commentary
This is yet another study pointing out the importance of blinding. Based on this and other similar studies it is our conclusion that studies or the results of studies without blinded assessors are grade U or at best grade B-U (see evidence-grading scale here).

1. Poolman RW, Struijs PA, Krips R, Sierevelt IN, Marti RK, Farrokhyar F, Bhandari M. Reporting of outcomes in orthopaedic randomized trials: does blinding of outcome assessors matter? J Bone Joint Surg Am. 2007 Mar;89(3):550-8. J Bone Joint Surg Am. 2007 Mar;89(3):550-8. PMID: 17332104. »

Return to Top.

Successful Evidence-based QI Project: Diabetes Management at Dreyer Medical Clinical
Example provided by Rami Rihani, PharmD, Director of Pharmacy

Delfini Introduction
Measuring clinical improvements is complex. One of the most important, frequently misunderstood issues is that cause and effect relationships can only be drawn with reasonable certainty from valid experiments (RCTs). However, if we have valid evidence from RCTs that an intervention leads to improved clinical outcomes, it is then reasonable to use process measures to evaluate the success of our evidence-based clinical improvement project.

Generally, we advise people to measure — not health status outcomes — but to perform a process measurement to evaluate the success of application of the intervention. In other words, we advise people to measure the success of implementation of the clinical improvement. For example, if we are trying to ensure patients get a beta-blocker post-MI, we would recommend looking to see if prescriptions increased for hospitalized MI patients — not to measure whether patient survival was improved. This is because observational data, such as information extracted from databases, can be highly prone to confounding. If health status outcomes are measured, then we advise people to ensure that there is a sufficient understanding of all those utilizing the data that conclusions drawn from observational data can be misleading. In the above example, if patient survival decreased, there could be many explanations.

However, if a health status outcome is measured, and if the before/after change is dramatic, it is reasonable to hypothesize that our project has been successful. For example…

Problem
Many diabetics have difficulty achieving a HbA1c <7.0. Frequently diabetics are told their HbA1cs are too high but active medication change is not aggressively pursued.

Evidence-based QI Project: A quality improvement group at Dreyer Medical Clinic developed a disease management initiative using PharmDs to actively titrate dosages of insulin and other drugs based on the Intermountain Health Care (IHC) diabetes management protocol. The process is as follows:

• Primary care physician (PCP) refers patient to the diabetes management program;
• PharmD aggressively titrates medication based on IHC protocol;
• PharmD monitors for safety and efficacy of medication interventions in collaboration with the PCP

Outcomes

Delfini Commentary
There was a significant improvement in the percent of patients achieving goal HbA1c and LDL associated with this project.

It is reasonable to believe that the clinical improvement project was successful. Using outcomes data from the UK Prospective Diabetes Study 35 (1), the QI team estimates that since inception, the disease management initiative resulted in the prevention of —

• four diabetes related deaths and
• nine microvascular events (defined as renal failure, death from renal failure, retinal photocoagulation, or vitreous hemorrhage)

1. Stratton, I,M., Adler, A.I., et al, Association of glycaemia with macrovascular and microvascular complications of type 2 diabetes (UKPDS 35): prospective observation study. BMJ 2000; 321; 405-12.

Return to Top.

Lack of double blinding is frequently found in surgical trials and results in uncertain evidence because of the problems stated above. A case study helps to illustrate this. A recent multicenter RCT, the Spine Patient Outcomes Research Trial (SPORT)[3] was a non-blinded trial that serves as an interesting case study of the blinding issues that arise when a surgical intervention is compared to a non-surgical intervention, and blinding is not attempted. The trial included patients with persistent (at least 6 weeks) disk-related pain and neurologic symptoms (sciatica) who were randomized to undergo diskectomy or receive usual care (not standardized but frequently including patient education, anti-inflammatory medication, and physical therapy, alone or in combination). There were a number of problems with this study including lack of power, poor control of non-study interventions, a high proportion of patients who crossed over between treatment strategies (43% randomized to surgery did not undergo surgery by 2 years and the 42% randomized to conservative care did receive surgery) and lack of blinding. The degree of missing data was 24%-27% without a true intention-to-treat analysis. Of great interest was an editorial that dealt with the problem of non-blinding in surgical studies. The editorialist, Flum, makes the following points [4]:

• While the technique of sham intervention is well accepted in studies of medications using inactive pills (placebos), simulated acupuncture, and nontherapeutic conversation in place of therapeutic psychiatric interventions, it has only occasionally been applied to surgical trials. This is unfortunate because the use of sham controls has been critical in understanding just how much patient expectation influences outcomes after an operation.
• A sham-controlled trial would be particularly relevant for spine surgery since the most commonly occurring and relevant outcomes are subjective.
• Patients chosing surgical options may have high expectations. They may include a higher level of emotional “investment” in surgical care compared with usual care based on the level of commitment resulting from a decision to have an operation and get through recovery. After the patient has accepted the risks of surgical intervention, the desire for improvement may drive perceptions about improvement.
• Patients who opt for surgery may also differ from patients who decline surgery in their beliefs regarding the benefits of invasive interventions.
• The surgeon’s expectations and direction are likely to play an important role in patient improvement.
• Given the proliferation of operative procedures for the treatment of subjective complaints like back pain, the need for sham controlled trials has never been greater.

• Example 1 — Ligation of Internal Mammary: After multiple observational studies suggesting that ligation of the internal mammary artery was helpful in patients with coronary disease, Cobb et al randomized patients to operative arterial ligation or a sham procedure. Both groups improved after the intervention, but there were similar, if not greater, improvements in subjective measures such as exercise tolerance and nitroglycerin use in the sham surgical group.
• Example 2 — Osteoarthritic Knee Surgery — and 3 — Osteoarthritic Knee Joint Irrigation: After multiple case series reported that patients with osteoarthritis of the knee improve after arthroscopic surgery, Moseley et al demonstrated just how much of that effect is related to the hopes, expectations, and beliefs of the patient. The investigators randomized 180 patients to undergo arthroscopy with debridement, arthroscopy with lavage, or sham arthroscopy. The power of expectation was strong and patients were unable to determine if they had been assigned to the treatment or sham groups— and all groups improved. At 2 years after randomization, all patients reported comparable pain scores and functional scores. Another sham-controlled study in patients with knee osteoarthritis demonstrated that patients benefit equally from irrigation of the joint and from sham irrigation.
• Example 4 — Parkinson’s Disease: Researchers found similar improvements in quality of life after direct brain injections of embryonic neurons or placebo in patients with advanced Parkinson’s disease.
• Example 5 — Transmyocardial Laser Revascularization in HF: Heart failure patients undergoing transmyocardial laser revascularization or sham procedures had equal improvements in subjective outcomes.
• Example 6 — Hernia: After hernia repair, there was equal improvement in pain control after cryoablation of nerves or sham interventions.
• Examples 7-9 — Laparoscopic Interventions: Multiple case series have reported benefit on subjective outcomes such as pain control, function, and readiness for discharge with laparoscopic cholecystectomy, colon resection, and appendectomy compared with conventional approaches..Bias arises when the clinical care team influences patient and discharge expectations though coaching, communication, and management. Randomized trials of these three procedures that included blinding of both the patients and the discharging clinicians to the treatment that patients received by placing large, side-to-side abdominal wall dressings demonstrate little or no difference in patients reaching discharge criteria. A reasonable conclusion is that when the clinician’s expectations and “coaching” were removed by placing a large bandage on the abdominal wall, the subjective benefits disappeared. Flum concludes that studies not addressing both patient and clinician expectation on subjective outcomes do not inform the clinical community about the true role of the intervention.

• Invasive procedures are associated with risks.
• There are great harms created by conducting studies that are of uncertain validity.
• Establishing community standards based on uncertain evidence is more likely to result in more harm than good.
• Sham-controlled trials are justified when uncertainty exists among clinicians and patients about the merits of an intervention.

2 Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of of treatment effects in controlled trials. JAMA 1995;273:408­12. PMID: 7823387.

3 Weinstein JN, Tosteson TD, Lurie JD, et al. Surgical vs nonoperative treatment for lumbar disk herniation: the Spine Patient Outcomes Research Trial (SPORT): a randomized trial. JAMA. 2006;296:2441-2450. PMID: 17119141

4 Flum DR. Interpreting Surgical Trials With Subjective Outcomes Avoiding UnSPORTsmanlike Conduct. JAMA, November 22/29, 2006—Vol 296, No. 20: 2483-1484. PMID: 17119146

Return to Top.

Blinding and RCTs

A recent article, Boutron I, Estellat C, Guittet L, Dechartres A, Sackett DL, et al. (2006) Methods of blinding in reports of randomized controlled trials assessing pharmacologic treatments: A systematic review. PLoS Med 3(10): e425. DOI: 10.1371/ journal.pmed.0030425, provides a great deal of useful information about and a way of classifying blinding in research studies. The authors evaluated blinding in RCTs of pharmacologic treatment published in 2004 in high impact-factor journals. The following are some key points from the article:

Delfini Commentary
Lack of blinding appears to be a major source of bias in RCTs. Just as well-done randomization and concealment of allocation to the study groups decreases the likelihood of selection bias, blinding of subjects and everyone working with the subjects or study data to the assigned intervention (double-blinding) decreases the likelihood of performance bias. Performance bias occurs when patients in one group experience care or exposures not experienced by patients in the other group(s) and the differences in care affect the study outcomes. Lack of blinding may affect outcomes in that:

• Unblinded subjects may report outcomes differently from blinded subjects, have different thresholds for leaving a study, seek (and possibly receive) additional care in different ways.
• Unblinded clinicians may behave differently towards patients than blinded clinicians.
• Using unblinded assessors may result in systematic differences in outcomes assessment (assessment bias).

A number of studies have shown that lack of blinding is associated with inflated treatment effects.

• Comparison of results of randomized and non-randomized studies across multiple interventions in multiple studies demonstrate that, in the majority of cases, observational studies are not consistent with the results of RCTs
• This study, using meta-epidemiological techniques, demonstrates that —
  • None of the study results can be adequately adjusted for bias in observational studies using historic and concurrent controls
  • Logistic regression on average increases bias when applied to observational studies

Conclusions

• Non-randomized studies may still give seriously misleading results even when those treated and control groups appear similar in key prognostic factors
• Standard methods of case-mix adjustment do not guarantee removal of bias
• Omission of important confounding factors can explain failure of adjustment as a substitute for randomization
• There is no known method for reliably adjusting for confounding factors in observational studies

Database Studies
Some groups have tried to demonstrate improved health outcomes (e.g., death, stroke, etc.) through studies of their databases. It should be remembered that this type of study is an observational study and prone to bias and confounding for the reasons explained above, plus it is highly prone to chance findings of statistical significance. Therefore, database studies may be useful for suggesting areas for further study, but they should not be thought of as valid studies from which cause and effect relationships can be concluded.

Return to Top.

Untrustable P-values & Abstracts

• Significant results in abstracts should generally be disbelieved
• Ongoing research has shown that more than 200 statistical tests are sometimes specified in trial protocols. If you compare a treatment with itself—that is, the null hypothesis of no difference is known to be true—the chance that one or more of 200 tests will be statistically significant at the 5% level is 99.996% if we assume the tests are independent
• Thus, the investigators or sponsor can be fairly confident that “something interesting will turn up.”
• Due allowance for multiple testing is rarely made, and it is generally not possible to discern reliably between primary and secondary outcomes
• Recent studies that compared protocols with trial reports have shown selective publication of outcomes, depending on the obtained P values, and that at least one primary outcome was changed, introduced, or omitted in 62% of the trials.
• The scope for bias is also large in observational studies. Many studies are underpowered and do not give any power calculations.
• Furthermore, a survey found that 92% of articles adjusted for confounders and reported a median of seven confounders but most did not specify whether they were pre-declared.
• Fourteen per cent of these articles reported more than 100 effect estimates, and subgroup analyses appeared in 57% of studies and were generally believed.
• The preponderance of significant results could be reduced if the following actions were taken.
  • First, if we need a conventional significance level at all, which is doubtful, it should be set at P < 0.001
  • Second, analysis of data and writing of manuscripts should be done blind, hiding the nature of the interventions, exposures, or disease status, as applicable, until all authors have approved the two versions of the text
• Third, journal editors should scrutinize abstracts more closely and demand that research protocols and raw data—both for randomized trials and for observational studies—be submitted with the manuscript.

In short, yet another reminder to read the methods section of papers and not rely on results or conclusions presented in abstracts.

• The first result in the abstract was statistically significant in 70% of the trials, 84% of cohort studies and 84% of case-control studies. Although many of these results were derived from subgroup or secondary analyses, or biased selection of results, they were presented without reservations in 98% of the trials