HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text (PDF) Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Gur, D. Articles by Hardesty, L. Search for Related Content PubMed PubMed Citation Articles by Gur, D. Articles by Hardesty, L.

On the Testing and Reporting of Computer-aided Detection Results for Lung Cancer Detection¹

¹ From the Department of Radiology, University of Pittsburgh and Magee-Womens Hospital, Imaging Research, Suite 4200, 300 Halket St, Pittsburgh, PA 15213-3180. Received December 11, 2003; accepted December 18. Address correspondence to D.G. (e-mail: gurd@upmc.edu).

Index terms: Cancer screening, 60.11, 60.1211, 60.1215 • Computers, diagnostic aid • Diagnostic radiology, observer performance • Editorials • Lung neoplasms, 60.30

There has been a great deal of interest in developing computer-aided detection (CAD) technology for early detection of lung cancer with both radiography and computed tomography (CT) (1–8). Studies focused on CAD alone or on the comparison of observer performance with and without CAD are reported with increasing frequency. There are many important issues to understand and consider in any evaluation of these efforts, and some of these issues are not easy to address. Two, however, can be.

First, the results of these studies are typically reported in terms of the sensitivity of the CAD scheme at a given false-positive cuing rate (eg, 80% sensitivity at two false-positive cues or marks per image or case). In these studies, sensitivity is relatively straightforward and easy to understand; the false-positive rate, however, is often reported in different units (eg, per image, lung, case, or examination), which makes it difficult to appreciate the absolute and relative performance levels of different CAD schemes (3,6,7). At times, even investigators at the same institution may be inconsistent in reporting results (4,5). This inconsistency in reporting is quite noticeable during presentations at national meetings, where, in a single session, several different units might be used for reporting false-positive rates (9–12).

To address this problem, we should all agree on one acceptable unit for reporting results. For example, we can agree to report the false-positive rate (number of false-positive cues or marks) per the entire CT lung scan instead of per image. Namely, the denominator is the full examination of the whole lung area as viewed and interpreted in a clinical screening environment. The advantages of this reporting method are that the meaning will be clear and that differences in imaging parameters such as section thickness will be inherently accounted for. Authors will have no need to explain why thinner or thicker sections result in different observations, as they would if the false-positive rate were reported per image or section.

The second issue is related to the cases used in these studies and is more difficult to address. Often, we do not know the outcome of these cases (at least, not at the time when imaging is performed); hence, we use an actionable indication as the target for CAD (12). Consensus by several radiologists is also frequently used to define positive cases with which to test CAD performance (5,6). Clearly, "actionable" and "consensus-based" are relative terms and represent subjective decisions, since what may be actionable to one radiologist in a specific environment may not be actionable to other radiologists in the same or different environments. Perhaps as important, it is not clear that what a radiologist considers actionable during the retrospective review of cases in a laboratory environment would be acted upon by the same radiologist in a clinical environment. Hence, our ability to compare results among studies, even in general terms, is substantially diminished. Despite the difficulty of obtaining the number of true-positive cases needed for CAD testing, it is important that we use, and report results based on, lung cancer cases that have been pathologically verified. Otherwise, we may unintentionally redefine the target of interest for the CAD scheme and the interpreter, and, ultimately, find many more actionable cases than there are or than we think there should be on the basis of the lung cancer incidence in a screening population or the ratio of positive to negative biopsy findings in an optimal practice environment.

In conclusion, we propose that all assessments of CAD schemes for lung cancer detection be based on cases in which imaging findings, particularly positive findings, have been pathologically verified, and that the results of CAD testing be reported in a standardized manner.

REFERENCES

1. Lo SC, Freedman MT, Lin JS, Mun SK. Automatic lung nodule detection using profile matching and back-propagation neural network techniques. J Digit Imaging 1993; 6:48-54.
2. Xu XW, Doi K, Kobayashi T, MacMahon H, Giger ML. Development of an improved CAD scheme for automated detection of lung nodules in digital chest images. Med Phys 1997; 24:1395-1403.
3. Kido S, Nakamura H, Ito W, Shimura K, Kato H. Computerized detection of pulmonary nodules by single-exposure dual-energy computed radiography of the chest. Eur J Radiol 2002; 44:198-204.
4. Giger ML, Bae KT, MacMahon H. Computerized detection of pulmonary nodules in computed tomography images. Invest Radiol 1994; 29:459-465.
5. Armato SG, Giger ML, MacMahon H. Automated detection of lung nodules in CT scans: preliminary results. Med Phys 2001; 28:1552-1560.
6. Brown MS, Goldin JG, Suh RD, McNitt-Gray MF, Sayre JW, Aberle DR. Lung micronodules: automated method for detection at thin-section CT—initial experience. Radiology 2003; 226:256-262.
7. Gurcan MN, Sahiner B, Petrick N, et al. Lung nodule detection on thoracic computed tomography images: preliminary evaluation of a computer-aided diagnosis system. Med Phys 2002; 29:2552-2558.
8. Zhao B, Gamsu G, Ginsberg MS, Jiang L, Schwartz LH. Automatic detection of small nodules on CT utilizing a local density maximum algorithm. J Appl Clin Med Phys 2003; 4:248-260.
9. Kim JH, Kim JS, Cho G, Bae KT. Automated detection of pulmonary nodules from multi-slice CT images of varying slice thickness and reconstruction interval (abstr). Radiology 2003; 229(P):617.
10. Enquobahrie A, Reeves AP, Yankeleviz DF, Kramer AL, Henschke CI. Automated detection of solid pulmonary nodules from low dose helical CT scans (abstr). Radiology 2003; 229(P):618.
11. Roy AS, Armato SG, Li F, Doi K, Sone S, Altman MB. Results of automated lung nodule detection organized by nodule type on a large database of low-dose CT scans (abstr). Radiology 2003; 229(P):618.
12. Wood SA, Schneider AC, Stapleton SJ, Castellino RA. The potential of computer-aided detection (CAD) for improving the detection of actionable lung nodules in multi-detector CT (MDCT) exams of the chest (abstr). Radiology 2003; 229(P):618.

This article has been cited by other articles:

				K Doi Current status and future potential of computer-aided diagnosis in medical imaging Br. J. Radiol., January 1, 2005; 78(suppl_1): S3 - s19. [Abstract] [Full Text] [PDF]

This Article Full Text (PDF) Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Gur, D. Articles by Hardesty, L. Search for Related Content PubMed PubMed Citation Articles by Gur, D. Articles by Hardesty, L.