Quality of Evidence:
Observational Studies

Contents      

• Cause & Effect Conclusions from Observations »
• More on the Problem with Drawing Cause-Effect Conclusions from Observational Studies »
• Bias in Observational Studies — More on HRT in Menopause »

We recently reviewed agreement between RCTs and observational studies dealing with treatment, having worked with so many folks who still believe that observational studies are sufficient for drawing reliable conclusions about effectiveness. This topic has become hot because of the HRT issue—in multiple well-conducted observational studies, authors reported ~40% ARR in CAD with HRT in secondary prevention, yet 5 valid RCTs showed no benefit.

Just to be clear, our take on this is that although an observational study may report outcomes that are similar to a RCT, you are going out on a limb if you conclude that any study type other than a RCT can establish cause and effect.

Here is what we found in our review, taking a historical perspective:

• In 1977 Chalmers et al. reported that a comparison of 6 RCTs to 8 observational studies of the use of anticoagulants in acute MI that the observational studies all showed larger benefit than did the RCTs (N Engl J Med.1977; 297:1091-6). He concluded that there are systematic biases in observational studies.

• In 1983 Chalmers reported that in 160 studies of 6 treatments in Cardiology, the outcomes in the intervention group were better than control in 60% of RCTs and 93% of observational studies. He again concluded that there are systematic biases in observational studies (N Engl J Med.1983;309: 1358-61).

• However, in 1998, Britton et al. concluded that there was not bias in observational studies after reporting that in 7 of 8 observational studies the outcomes were similar to outcomes in RCTs (Health Technol Assess. 1998; 2:1-124).

• In 2000 Guyatt et al. reviewed 13 RCTs and 17 observational studies of pregnancy in adolescence and reported that 6 of 8 outcomes of observational studies showed significant benefit, but that none of the RCTs showed significant benefit. Guyatt’s conclusion was that treatment decisions should be based on observational studies ONLY when RCTs are unavailable and ONLY with careful consideration of possible biases (Journal Clinical Epidemiology.200; 53: 167-174).

• However, using a similar approach to the one used above, two authors (Benson and Concato) in 2000 concluded that observational studies usually provide valid information. An editorial (Ioannidis JP, Haidich A, and Lau J. N Engl J Med. 2000;342: 879-880) stated in response to these authors (and other authors who conclude that cause-effect conclusions can be drawn from observational studies) that:

• Benson Concato et al are still dealing with only a very small portion of randomized and observational research.
• Their sampling failed to capture some prodigious discrepancies between the two methods. Interventions such as beta carotene and tocopherol, which have brought fame to observational epidemiologists, crashed when they were tested in rigorous randomized controlled trials.
• Given the hundreds of thousands of trials and observational studies that have been conducted and are still being conducted, the number of topics studied in the two reports is limited and subject to strong selection biases.
• We should not abandon plans for RCTs in favor of quick and dirty observational designs.

Bottom Lines

• Randomization is the only effective means of controlling for known and unknown confounders.
• Even with RCTs, threats to validity remain—for example:
  • Trials with inadequate or unclear concealment of allocation show more beneficial effects than adequately concealed trials;
  • Open trials tend to show more beneficial effects than double-blind trials (Health Technology Assessment 2003; Vol.7: No.1).
• Be cautious about accepting the reported treatment effects from observational studies because even well-conducted observational studies may not provide valid evidence because of bias inherent in observational studies
  • Even valid observational studies may provide you with misleading results – even going so far as to suggest benefit when in fact the actual result may be harms as we have seen with HRT.
  • Even if observational studies are subsequently found to have “agreed” with valid RCTs, they may overestimate treatment effects when compared with RCTs
• Always look for RCTs and always look at the methods.
  • RCTs without blinding may overestimate treatment effects (or may provide you with misleading results).
  • One SR estimates that clinical trials without adequate concealment of allocation produce estimates of effect that are on average 40% larger than clinical trials with adequately concealed random allocation (Kunz and Oxman, BMJ 1998; 317: 1185-90).
  • The bias, however, can go in either direction.
• Be cautious about accepting the treatment effects of low quality RCTs and non-randomized clinical trials because they may provide inaccurate estimates of treatment effects.

So for those who point to the sometimes agreement between the two study types, we quote one of our friends who points out that even a stopped clock is right at least twice a day.

Our last teaching engagement was in Framingham, Massachusetts and reminds us of the value of observational studies to assist us in developing risk stratification models. The Framingham Study began in 1948 as the first prospective study of cardiovascular disease and is important because through observations it has identified cardiovascular disease (CVD) risk factors which can be associated with morbidity and mortality.

But there is good evidence that basing cause and effect conclusions from observational studies is unreliable. Cause and effect conclusions should be based on randomized controlled trials (RCTs) where bias, confounding and chance have been ruled out as possible explanations for the observed association between the intervention and the outcome. Because there are so many observational studies published each week and because we keep seeing health professionals inappropriately basing treatment decisions on them, it is worthwhile summarizing an excellent review of the literature on this topic.

The study and literature review can be found in the reference:

Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F, Petticrew M, Altman DG. Evaluating non-randomised intervention studies. Health Technology Assessment 2003; Vol. 7: No. 27.

Some key points from this article:

• Comparison of results of randomized and non-randomized studies across multiple interventions in multiple studies demonstrate that, in the majority of cases, observational studies are not consistent with the results of RCTs
• This study, using meta-epidemiological techniques, demonstrates that —
  • None of the study results can be adequately adjusted for bias in observational studies using historic and concurrent controls
  • Logistic regression on average increases bias when applied to observational studies

Conclusions

• Non-randomized studies may still give seriously misleading results even when those treated and control groups appear similar in key prognostic factors
• Standard methods of case-mix adjustment do not guarantee removal of bias
• Omission of important confounding factors can explain failure of adjustment as a substitute for randomization
• There is no known method for reliably adjusting for confounding factors in observational studies

Delfini Commentary
Extreme caution is urged when considering results of observational studies in interventions for screening, prevention and therapy. Cause and effect conclusions should only be drawn from RCTs.

One reason for this is that there may be major differences in the characteristics (prognostic factors) of individuals who choose a therapy compared to people who do not choose that therapy. A classic example is hormone replacement therapy after myocardial infarction (MI) in women. Most observational studies reported that roughly twice as many women who did not choose to take hormone replacement therapy (HRT) had a recurrent MI compared to women who chose to take HRT. This led people to believe — incorrectly — that HRT caused this benefit. Later, well-done RCTs were conducted and no such benefit was found. Why? The most likely reason is that the observational studies were highly prone to biases resulting from differences between the groups which could not be eliminated even with statistical adjustments in which researchers try to balance confounders between the groups, such as adjusting for smoking.

Another reason is that in observations, investigators do not “control” all elements of the study as they do in RCTs. The end result is that in observational studies other aspects affecting the groups are almost certain to be different in important ways which are likely to explain or affect the study results.

Key Point
Any difference between groups — except for what is being studied (e.g., HRT use) — is a bias.

In the case of HRT after MI, selection bias was present in that women who chose to take HRT were probably more likely to be “health-conscious,” exercise, watch their diets, etc., making them different from the women who did not take HRT. It is also likely that there were other differences in how the two groups experienced their health care because in observational studies there is no formal protocol and so there will be differences in many ways that could affected observed outcomes such as other therapies used, how outcomes are assessed, frequency for follow-up, and so on.

Even with statistical adjustments for differences between potential and known prognostic characteristics of the groups, bias cannot be reliably eliminated because whatever is actually responsible for the outcome (i.e., the confounder) is what would have to be adjusted. This would entail having advance knowledge of cause and effect (but that is why the study is being conducted). Plus statistical adjustment has limitations. How could every single factor that made the HRT users different be adjusted? Humans embody an infinite number of variables such as characteristics and exposures.

Comparisons of RCTs and observational studies of the same interventions have repeatedly demonstrated that even with the most meticulous statistical adjustments, bias cannot be reliably eliminated from observational studies. The key message is that without randomization and assurance that interventions and assessments are the same for both study and comparison groups, one cannot reliably draw conclusions about cause and effect relationships. Associations between interventions and outcomes in observational studies are very likely to be due to bias or confounding. Therefore, observational studies are only useful for hypothesis-generating when considering questions of preventive, screening or therapeutic interventions.

Database Studies
Some groups have tried to demonstrate improved health outcomes (e.g., death, stroke, etc.) through studies of their databases. It should be remembered that this type of study is an observational study and prone to bias and confounding for the reasons explained above, plus it is highly prone to chance findings of statistical significance. Therefore, database studies may be useful for suggesting areas for further study, but they should not be thought of as valid studies from which cause and effect relationships can be concluded.

After RCTs (e.g., WHI and HERS) reported no benefit of hormone replacement therapy in preventing coronary heart disease when compared to placebo, many authors explained the dramatic difference from (presumed) benefit reported in observational studies on the basis of the “healthy user effect” a type of selection bias. The conclusion was that in the observational studies women who elected to take HRT were different (i.e., healthier) than those who did not seek out HRT.

In the Dec. 2 Annals of Internal Medicine, SG Pauker points out there are other potential differences between the RCTs and observational studies that might explain the dramatic differences in reported coronary artery disease outcomes between observational studies and RCTs:

1. In the Nurses Health Study (observational study) silent MI was not studied.
2. HRT users who believed that HRT protected against coronary heart disease (subjects in the observational studies) might not interpret ischemic pain as related to their hearts and not seek care. This illustrates why blinding of study subjects is so important.
3. HRT users who believed that HRT protected against coronary heart disease and with atypical ischemic pain might describe their symptoms in a way that was interpreted as non-cardiac by their physicians—again illustrating the importance of blinding.
4. Physicians completing death certificates might also believe that HRT protects against CHD and assign the cause of death to a condition other than CHD—an assessment bias.

This article is a nice reminder that in observational studies there is always potential for selection bias, performance bias and assessment bias.

Abstract:
http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=
Retrieve&db=PubMed&list_uids=14644895&dopt=Abstract