DelfiniClick™

Quality of Evidence:
Screening & Diagostic Studies

newest
02/23/08: The Bias of “Survival” in Cancer Screening Studies: CT Screening for Lung Cancer in Smokers—Back-to-Basics for Validity »

Contents      

  • The Bias of “Survival” in Cancer Screening Studies: CT Screening for Lung Cancer in Smokers—Back-to-Basics for Validity »
  • newest Bias in Diagnostic Studies »
The Bias of “Survival” in Cancer Screening Studies: CT Screening for Lung Cancer in Smokers—Back-to-Basics for Validity
Readers should be aware that survival as an outcome measure for cancer screening studies is always biased due to lead-time bias. For any screening studies, readers should consider the opportunities for the play of lead-time, length, volunteer and overdiagnosis bias. Consider the following: 

Case
A middle-aged smoker is concerned about developing lung disease or lung cancer. He has just read a newspaper report that a new screening scan for lung cancer may prevent death from lung cancer. He knows that lung cancer is the number one cancer killer of Americans and decides to “Google” the topic of lung cancer using the following search terms: “lung cancer screening Seattle.” He clicks on a local cancer institute at the top of the Google hit list and reads, “Innovative technology offers new hope. Low Dose Spiral Computerized Tomography (CT) lung scanning can detect tiny lung abnormalities long before they show up on conventional X-rays. The scan is painless, takes only a few moments of your time and may help save your life.” He reads further and finds it will cost him $300 due at the time of the appointment and that it is unlikely his insurance will pay for the scan. Should he schedule the scan?

On October 26, 2006, the New England Journal of Medicine’s lead article from the International Early Lung Cancer Action Project (I-ELCAP) created renewed interest in lung cancer screening using low-dose CT or “spiral CT” scanning.1 The I-ELCAP study was described by the authors in the abstract as “a large collaborative study” of lung cancer screening in high-risk subjects. The authors reported a 10-year survival rate for screened patients of 88%. 7 of the 10 highest-circulation newspapers in the country carried the “breakthrough” story as did the major wire services and 4 of the 5 major television networks. The Lung Cancer Alliance, an advocacy group for people with lung cancer launched an advertising campaign encouraging the public to get screened for lung cancer. Headlines similar to the following appeared all over the country: “Landmark Study Reveals that Lung Cancer 10-Year Survival Dramatically Improves with Annual CT Screening and Prompt Treatment.” The American Cancer Society wrote a very positive review of this study which is still on their website as of this writing.

Validity of I-ELCAP
Whoooh, Nellie! The I-ELCAP study was a large case series of spiral CT lung cancer screening—a very low quality study design. The study involved 31,567 asymptomatic persons at risk for lung cancer because of their history of cigarette smoking, occupational exposure (e.g., to asbestos, beryllium, uranium, or radon) or exposure to secondhand smoke with or without a family history of lung cancer.

Many of the key threats to validity were pointed out in the Feb. 15, 2007 issue of the New England Journal of Medicine’s Correspondence section. Later a commentary, very critical of I-ELCAP, was published in the Archives of Internal Medicine.2

The following threats to validity were pointed out by the NEJM correspondents and the scientists writing in the Archives of Internal Medicine along with our comments

  • Lack of Comparison Group: There was no control group (comparison group). A comparison group is a requirement in studies of effectiveness of new interventions in order to be considered high quality science. (In this case, an appropriate study would be a well-designed randomized controlled trial (RCT) comparing a group of smokers screened by spiral CT with a group of smokers who did not receive CT screening.
    Comment: several RCTs of lung cancer screening with low-dose CT are being currently being conducted but results are not expected until 2009.)
  • Survival as an Outcome Measure is Biased in Cancer Screening Studies: I-ELCAP used survival as its outcome measure, which in cancer screening studies is biased. The study reports survival rates much higher that those reported in previous studies. Why was the reported survival so much higher in I-ELCAP than reports of survival made by groups such as the National Cancer Institute? Cancer “survival” is always lengthened by early detection, even when deaths are not delayed and even when no lives are actually saved by finding cancers early. First, even though people may die at the same time as if they had not been screened, screening allows researchers to count more years of living with the diagnosis of lung cancer as increased survival because of the earlier diagnosis. This type of bias (lead-time bias) occurs because people are diagnosed earlier with screening, but they die at the same time they would have died without the early detection. This is a well-known screening bias present in many cancer screening studies. Second, screening picks up cancers that would never have caused death (this is referred to as overdiagnosis bias) or cancers that would have progressed very slowly (length bias).

    Comment: There is an additional bias that can come into play in screening studies: patients who volunteer for screening may be “different” (healthier) from the larger population and may have a better prognosis (volunteer bias).

  • Harms of Screening: people should, before undergoing CT screening for lung cancer, know that many people will have false positive tests. In U.S. and European clinical practices, approximately half of patients undergoing surgical biopsy of indeterminate nodules subsequently receive a benign diagnosis.3 To find out if the nodule is benign involves risk. The United States Preventive Service Task Force’s (USPSTF) 2004 review of spiral CT scanning for lung cancer reported the risks of screening: overall, mortality rates for lung biopsy using thoracotomy and tissue resection range from 1.3% to 11.6% and morbidity rates from 8.8% to 44% with higher rates usually being associated with greater amounts of tissue resection.4 Additionally, there is a potential risk of CT screening because of the radiation. There may be harms which show up much later to chest organs and tissues, including damage to the heart muscle or coronary arteries from the CT radiation.

A more recent article provides a very negative perspective on the value of CT screening for lung cancer. Based on a comparison of the predicted with the observed number of new lung cancer cases, lung cancer resections, advanced lung cancer cases and deaths from lung cancer, Bach et al.5 suggest that annual CT screening does not result in a decline in the number of advanced cancers detected or death from lung cancer, but does result in a 10-fold increase in lung cancer surgeries resulting from screening. The implication is that there will be huge increases in surgical operations, and harms might well outweigh benefits from CT screening. The authors cite US postoperative mortality rates for lung cancer surgery as averaging about 5% and the frequency of serious complications ranging from 20% to 44%. They conclude that, “Until more conclusive data are available, asymptomatic individuals should not be screened outside of clinical research studies that have a reasonable likelihood of further clarifying the potential benefits and risks.”

Criteria for Screening Tests and Spiral CT
Frame and Carlson developed criteria for a screening test in 1975.6 Similar criteria are now used by many health care organizations and by the USPSTF. Let’s compare the evidence for spiral CT as a screening test with these criteria.

FRAME AND CARLSON CRITERIA FOR EVALUATION A SCREENING TEST
1. The disease must have a significant effect on quality or quantity of life.
Yes. Lung cancer is the major cause of cancer deaths in the U.S.

2. Acceptable methods of treatment must be available.
No. No valid RCTs have demonstrated improved outcomes (morbidity or mortality) with screening. RCTs which may answer this question may be available in 2009.

3. The disease must have an asymptomatic period during which detection and treatment significantly reduce morbidity and/or mortality.
No. This appears to be a major issue. RCT outcome data demonstrating improved outcomes with CT screening are needed. In the past, multiple studies of chest X-ray screening for lung cancer reported detection of increased numbers of small early stage lung cancers but mortality was not reduced by the screening and the chest X-ray screening was for the most part abandoned. Lead-time, length and overdiagnosis bias (described above) were probably present in those studies.

4. Treatment in the asymptomatic phase must yield a therapeutic result superior to that obtained by delaying treatment until symptoms appear.
No RCTs have shown this. See number 3 above.

5. Tests must be available at a reasonable cost to detect the condition in the asymptomatic period.
No. Tests are expensive and not reliable for detecting early lung cancer that will benefit from surgery or other treatment at the detection time. There is no currently available high quality science to tell us how many false positive and false negative tests will occur with screening. Because of lead, length overdiagnosis and volunteer biases along with lack of RCTs showing improved outcomes with screening, it is not known if detecting asymptomatic lung cancer improves meaningful clinical outcomes. Cost considerations must also include further testing that will result from positive screening CTs (e.g., serial imaging, thoracotomy, lung resection with significant costs of the resulting morbidity and mortality).

6. The incidence of the condition must be sufficient to justify the cost of screening.
Although the cost is high, the incidence is so high that if screening and treatment were shown to be effective, cost may not be a significant barrier.

In addition to these criteria, the screening test itself must be validated with attention to sensitivity, specificity, accuracy and precision.
CT screening appears to have high rates of "false positive" findings as noted above. Until valid studies comparing screening to no screening with low-dose CT scans are available, we will lack valid information on sensitivity, specificity, predictive values, accuracy and precision and, thus, will be unable to predict outcomes for patients at risk.

The Bottom Line—What Should We Tell Patients?
Let’s go back to our case study. What should we advise our patient about spiral CT screening? There are two take-home messages for patients:

1. Stop smoking or don’t start. It is the best way to prevent dying from lung cancer.

2. The science is insufficient to assure you that any screening test is going to find lung cancer that will, with treatment, clearly result in greater benefits than harms. A "positive" result of screening may represent a non-lethal nodule (false positive test) that will require more tests or an operation on your chest. Chest surgery for positive findings has significant risks and unproven benefits. There is also the possibility that you will be falsely reassured if you have a “negative” result. A negative CT scan is no guarantee that you are protected from lung cancer. That’s why knowledgeable groups do not recommend screening for lung cancer. For example --

  • The US Preventive Services Task Force states, “Evidence is insufficient to recommend for or against screening asymptomatic persons for lung cancer with either low dose computerized tomography, chest x-ray, sputum cytology, or a combination of these tests (2004).”
  • The American College of Chest Physicians recommends “against the use of low dose CT, chest radiographs, or sputum cytology for lung cancer screening, including smokers or others at high risk, except in the context of a clinical trial (2007).”

Health care professionals initiated lung cancer screening programs using chest radiographs based on studies with the same biases mentioned above. Let's not go down this road again.

References
  1. International Early Lung Cancer Action Program Investigators, Henschke CI, Yankelevitz DF, Libby DM, Pasmantier MW, Smith JP, Miettinen OS. Survival of patients with stage I lung cancer detected on CT screening. N Engl J Med. 2006 Oct 26;355(17):1763-71. PMID: 17065637
  2. Welch HG, Woloshin S, Schwartz LM, Gordis L, Gøtzsche PC, Harris R, Kramer BS, Ransohoff DF. Overstating the evidence for lung cancer screening: the International Early Lung Cancer Action Program (I-ELCAP) study. Arch Intern Med. 2007 Nov 26;167(21):2289-95. PMID: 18039986
  3. Swensen SJ, Jett JR, Sloan JA, et al. Screening for lung cancer with low-dose spiral computed tomography. Am J Respir Crit Care Med 2002;165(4):508-13.
  4. Humphrey LL, Teutsch S, Johnson MS. Lung Cancer Screening with Sputum Cytologic Examination, Chest Radiography, and Computed Tomography: An Update for the U.S. Preventive Services Task Force. Ann Intern Med 2004;140:740-53. http://www.ahrq.gov/clinic/3rduspstf/lungcancer/lungsum.htm. Accessed 1/30/08.
  5. Bach PB, Jett JR, Pastorino U, Tockman MS, Swensen SJ, Begg CB. Computed tomography screening and lung cancer outcomes. JAMA. 2007;297(9):953-961.
  6. Frame PS, Carlson SJ. A critical review of periodic health screening using specific screening criteria. J Fam Pract 1975;2:29-36, 123-9, 189-94, 283-9.
Bias in Diagnostic Studies

In reviewing the literature on diagnostic testing, we recently came across an article we found very useful in further demonstrating the effect bias frequently has on making results more impressive than they actually are.

The authors evaluated 184 original studies of 218 diagnostic tests and found that only 6.8% of the 218 evaluations met all 8 criteria of a good diagnostic test. It appears that studies evaluating diagnostic tests in a diseased population and a separate control group (as compared to studies that used subjects from a relevant clinical population) overestimated the diagnostic performance of the test by a factor of 3.

Studies in which different reference tests were used for positive and negative results of the test under study overestimated the diagnostic performance of the diagnostic test (compared with studies using a single reference test for all patients) by a factor of 2.2.

Diagnostic performance was also overestimated when the reference test was interpreted with knowledge of the test result (relative odds ration, 1.3), when no criteria for the test were described (relative odds ratio, 1.7), and when no description of the population under study was provided (relative odds ratio, 1.4).

The authors conclude that diagnostic studies with methodological shortcomings may overestimate the accuracy of a diagnostic test, particularly those including nonrepresentative patients or applying different reference standards.

Reference and abstract:

Lijmer JG et al. Empirical Evidence of Design-Related Bias in Studies of Diagnostic Tests, JAMA. 1999;282:1061-1066. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=
Retrieve&db=pubmed&dopt=Abstract&list_uids=10493205


© Delfini Group, LLC, 2002-2008. All Rights Reserved Worldwide.
Use of this website implies your agreement to our Notices »

Counter courtesy of www.amazingcounters.com and sponsors below:

Web Site Hit Counters
Dell Computer

Notebook Computer
Notebook Computer