| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Computer Applications |
1 Department of Radiology, Beth Israel Deaconess Medical Center and Harvard Medical School, 330 Brookline Ave, Boston, MA 02215.
 |
Abstract |
|---|
 TOP Abstract IntroductionMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
MATERIALS AND METHODS: During 8 months, 80 patients underwent pelvic US at a community-based outpatient site. Images were electronically transmitted to a remote medical center as they were acquired at the community site and were printed on a laser printer identical to the one used at the outpatient site. The reference standard for correct diagnosis was based on histopathologic findings (n = 13), additional imaging results (n = 34), or review by a second independent observer (n = 33). Both an on-site and a remote radiologist interpreted the images, and their interpretations were rated as agree, both correct; agree, both incorrect; or disagree. Cases of disagreement were rated as major or minor.
RESULTS: On-site and remote radiologists agreed in 69 of 80 patients (86%), and both radiologists were correct in all of these cases. There were 10 minor discrepancies and one major discrepancy. The diagnostic accuracies of the on-site and remote radiologists were 92% and 94%, respectively.
CONCLUSION: High levels of diagnostic accuracy can be achieved by radiologists interpreting static US images. Strict protocols and excellent communication between the radiologist and sonographer are necessary to avoid diagnostic errors.
Index terms: Pelvic organs, diseases, 85.31 • Pelvic organs, US, 85.12981, 85.12983 • Pregnancy, US, 85.131 • Radiology and radiologists, departmental management • Teleradiology
|
Introduction |
|---|
|
TOPAbstractIntroductionMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
With imaging procedures that produce static images, such as computed tomography (CT) or magnetic resonance (MR) imaging, there is virtually no loss of information when images are transferred to a distant site. Remote interpretation of static ultrasonographic (US) images poses an additional challenge, however, because the potential information provided by real-time imaging is lost when only static images are available for interpretation. In addition, if static images are collected during the day and are "batch processed" at the end of that day, radiologists cannot obtain additional views (1). To overcome these limitations, the authors of several recent articles have advocated the use of telesonography (2) with real-time transmission of US images through various methods of video transmission (3–5). Even so, transmission of real-time US images requires the installation of fiberoptic cable and video hardware.
The purpose of our study was to compare the diagnostic accuracy of radiologists interpreting static US images electronically transmitted to our academic medical center (remote radiologists) with the accuracy of radiologists performing "hands-on" US at our department's community-based outpatient site (on-site radiologists). Pelvic US was used, because these studies are performed for a relatively limited set of indications in a relatively homogeneous patient population, and only one organ system is involved.
|
MATERIALS AND METHODS |
|---|
|
TOPAbstractIntroductionMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
All US images were acquired at the outpatient center by using a model HDI 3000 (ATL Ultrasound, Bothel, Wash) or 128XP (Acuson, Mountain View, Calif) US unit. The images were printed with a laser printer (Ektascan 2180; Eastman Kodak, Rochester, NY). All images were printed on 14 x 17-inch film in a four-by-five–image format (20 images per sheet of film) with a 512 x 512 matrix. Each US unit was fitted with an Image Acquisition Unit (Eastman Kodak) that served as a "frame grabber" and captured 8-bit black-and-white images or 24-bit color images. The spatial resolution of the frame grabber was dependent on the resolution of the individual US units. For the ATL unit, the spatial resolution was 640 x 476; for the Acuson unit, the spatial resolution was 750 x 480.
The on-site radiologist had the opportunity to perform US in each patient after the sonographer (J.M.C., D.L.W.) had obtained images according to our departmental protocols. When the study was complete, the on-site radiologist recorded his or her diagnosis and level of certainty. Level of certainty was rated on a five-point scale, with a score of 1 for uncertain, 2 for somewhat uncertain, 3 for somewhat certain, 4 for moderately certain, and 5 for very certain.
As images were obtained at the outpatient center, they were simultaneously transmitted via a T1 line to an identical laser printer at our academic medical center where they were printed by using the same format as that used at the outpatient center. The T1 line used to transmit the static US images was also used to transmit data between the outpatient center and the academic medical center's hospital information system and radiology information system. Therefore, additional network capabilities were not needed to transmit the US images. The average transmission time for a single sheet of film (20 images) was 3–5 minutes, depending on network traffic.
The remote radiologists (including D.L. and C.R.M.) had either completed a fellowship in US (n = 3) or devoted more than 75% of their clinical time to imaging in women (n = 1). After the images were transmitted to the academic center, the sonographer attempted to call the remote radiologist and discuss the case. The remote radiologist had the opportunity to ask the sonographer to obtain additional views, but unlike the on-site radiologist, he or she did not have the opportunity to perform scanning in the patient. When the remote radiologist had received all of the images that he or she deemed necessary, the diagnosis and level of certainty were recorded. Cases were received during the usual workday at the academic center; because of the volume of work at the academic department (where approximately 40–50 US procedures are performed each day), images in 14 of the study patients were not interpreted at the time of the examination.
Differences in the level of certainty between the two radiologists for each case were compared by using a two-tailed 
2 test. The diagnoses provided by the on-site radiologist and by the remote radiologist were also compared by an independent observer (L.F.) who was blinded to the source of the data. Agreement between the on-site radiologist and the remote radiologist was categorized as agree, both correct; agree, both incorrect; or disagree. Cases of disagreement were graded as major (having a serious effect on the patient's subsequent care) or minor (likely to have no effect on the patient's subsequent care). Minor discrepancies were further categorized as resulting from errors of observation or errors of interpretation. Less specific diagnoses were considered to be minor errors of interpretation.
In 47 patients, the reference standard for the correct diagnosis was established on the basis of histopathologic findings (n = 13) or follow-up with additional US or other imaging studies (n = 34). In 33 patients, pathologic findings or follow-up images were not available. The results in these 33 patients were reviewed by a second independent observer (C.A.H.) who was fellowship trained in US. This observer recorded the diagnosis without previous knowledge of the original interpretation of the US image. In these 33 patients, the diagnosis of this independent observer was considered to be the reference standard.
|
RESULTS |
|---|
|
TOPAbstractIntroductionMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
In 10 patients, there were minor discrepancies between the interpretation of the remote radiologist and that of the on-site radiologist, as detailed in Table 1. In five of these patients, the remote radiologist provided the more accurate diagnosis; in the other five, the on-site radiologist provided the more accurate diagnosis. If both major and minor discrepancies are considered to be incorrect, the diagnostic accuracies of the remote radiologists and of the on-site radiologists were 92% (74 of 80) and 94% (75 of 80), respectively.
|
|
|
DISCUSSION |
|---|
|
TOPAbstractIntroductionMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
Our intent was to study the performance of an existing technology during actual clinical use. To this end, we sacrificed some purity of study design in that two variables (technology and physician training) were altered at each site. The technology variable was that real-time US was available to on-site radiologists at the community site but that only static images were available to remote radiologists at the academic medical center. The physician training variable was that on-site radiologists did not have subspecialty training in US, unlike the remote radiologists practicing at the academic medical center. The outcome that we studied—differences in diagnostic accuracy in relationship to differences in technology—would have been measured more accurately if the training of the radiologists at both sites had been equal. In actual clinical practice, however, remote interpretation of images is usually used to deliver the expertise of subspecialty-trained radiologists to community-based sites.
Among the 80 patients in our study, we found one major discrepancy between interpretations by the two radiologists. This discrepancy was not caused by the remote radiologist missing specific anatomic findings but rather was due to the remote radiologist interpreting the US findings in an inappropriate context. This case involved a patient in whom the appropriate measurements of a first-trimester pregnancy were made (6 weeks 4 days), but the remote radiologist failed to interpret the measurements in the context of the gestational age (8 weeks). In addition, the remote radiologist did not comment on the abnormally slow fetal heart rate (74 beats per minute). In actual practice, we expect that communication between the sonographer and the radiologist would have been better and that both of these omissions would not have occurred. In addition, without the benefit of real-time scanning, the effect of the fetal bradycardia may have been diminished and, hence, overlooked by the remote radiologist interpreting the static US images.
Both of these "misses" could easily be corrected by improving the systems involved with remote US interpretations. For example, a checklist could be developed for the remote radiologist reading static US images. This checklist could be specifically tailored to each type of US examination (obstetric, pelvic, vascular), which should ensure that remote radiologists attend to features of the US image that may be less apparent when interpreting static rather than real-time images. This checklist could easily be implemented by using a standard fax machine. We envision that when the sonographer (working at the community site) is finished with the US examination, he or she would complete a simple worksheet in which relevant positive and negative findings are indicated in a "check the box" or "fill in the blank" format. The sonographer would then fax the worksheet to the remote radiologist. A separate worksheet could also be developed for the common types of outpatient US examinations (eg, abdominal, pelvic, obstetric). Implementation of this system would require only a standard fax machine at the community and the academic sites, and it would be much easier to implement than a videoconferencing system over a computer network.
Minor errors of observation were made in six patients, but the correct diagnosis was made by the on-site radiologist in four of these patients. These errors in observation were likely made by the remote radiologist because of his or her inability to perform US in the patient and to resolve subtle findings in real time. For example, in one case, the question of a thickened nuchal fold might have been resolved by performing imaging in that patient with the fetus in a different position. Alternatively, this may have been a valid finding in a normal fetus with nuchal thickening. The other two cases in which the remote radiologist was not correct involved patients in whom the endometrium was slightly thickened. The ability to resolve the margins of the endometrium in real time might have helped the remote radiologist make the correct diagnosis.
The fact that the second, independent observer had only static images available for interpretation adds some bias to our study. For example, in one case, the on-site radiologist thought there were small fibroids in the uterus, which were not appreciated by the remote radiologist or by the independent observer. In another case, the remote radiologist and the independent observer noted a small ovarian calcification and recommended follow-up to evaluate for a possible dermoid cyst. The inhomogeneity of the echotexture of the uterus may have been beyond the resolution of the static images, and the calcification seen in the ovary with the suspected dermoid cyst may have been the result of a shadowing artifact from adjacent bowel gas. Without the benefit of confirmation with another imaging modality, the interpretation of these findings by the independent observer must be accepted in its limited context.
In four patients, there was a discrepancy attributed to errors of interpretation between the remote radiologist and the on-site radiologist. In three of these patients, the on-site radiologist noted all of the essential findings, but the remote radiologist provided a more accurate interpretation. This likely reflects the additional training and expertise of the remote radiologist in interpreting US images. In one patient with cystadenoma, however, the remote radiologist, who suggested an endometrioma as the cause, was less correct than the on-site radiologist, who described a multiseptated cyst. Even so, both radiologists described a non–self-limiting process that necessitated additional evaluation.
Remote radiologists were more likely than on-site radiologists to rate their level of certainty as very certain. There was no statistically significant difference, however, in the frequency with which remote or on-site radiologists rated any level of certainty. We did not address the effect that the radiologist's level of certainty may have had on the clinical practice of the referring physician. It is possible, however, that the added confidence exhibited by the remote radiologists may lead to the performance of fewer follow-up tests.
One major limitation of the system described in this article was that the remote radiologist practiced in a busy US department and often found it burdensome to retrieve images from the laser printer and establish telephone contact with the sonographer at the time of the examination. Because the interpretation provided by the remote radiologist was not used for patient care, both the remote radiologist and the sonographer exerted less effort to pursue this contact than would occur in clinical practice. When our remote US system is implemented in clinical practice, additional resources will be available at our academic site to address these issues.
A second limitation of our study was that we evaluated only the performance of remote radiologists when interpreting pelvic US studies. We do not know if our results can be generalized to other types of US studies.
A third limitation was the lack of follow-up in 33 patients. We attempted to address this by having a second independent observer, who was fellowship trained in US, review all cases in which follow-up was not available. The interpretation of this observer served as the reference standard for these 33 patients. Although this observer may have been wrong, we have no reason to suspect this was the case. In addition, a discrepancy between the interpretations of the on-site and the remote radiologist occurred in only five of 33 patients (15%) for whom the independent observer's interpretation served as the reference standard. All five discrepancies were minor and were unlikely to have affected patient care. The other 28 patients for whom the independent observer's interpretation served as the reference standard were all those in whom the interpretations of the on-site radiologist and of the remote radiologist agreed.
In conclusion, high levels of diagnostic accuracy can be achieved by specialty-trained radiologists performing remote interpretations of static US images. By reviewing static images before the sonographer completed the US examination, we eliminated many of the problems encountered when static images are acquired by a sonographer and then batch read by a radiologist at the end of the day. In 14 of 80 patients (18%), the remote radiologist requested that additional views be obtained. For a system in which the images are batch read at the end of the day, this may have resulted in a patient callback rate of 10%–20%. Alternatively, the radiologist may have been induced to interpret the images as being "technically limited" and hedged his or her diagnosis. Such limited reports may result in requests for additional imaging studies and diminish the credibility of the radiologist.
Some of the advantages of our system include immediate feedback to the sonographer, the ability to obtain additional views, and more timely reporting of US findings to the referring clinician. As with any remote imaging system, the success of our project relies heavily on the ability of the sonographer, the establishment of stringent quality-control mechanisms, and excellent communication between the radiologist and the sonographer.
|
Footnotes |
|---|
Address reprint requests to M.P.R.
Author contributions: Guarantor of integrity of entire study, M.P.R.; study concepts and design, M.P.R., D.L.; definition of intellectual content, M.P.R., D.L.; literature research, M.P.R.; clinical studies, M.P.R., D.L., J.M.C., L.F., C.A.H., D.L.W., C.R.M.; data acquisition and analysis, M.P.R., L.F., C.A.H.; statistical analysis, M.P.R.; manuscript preparation and editing, M.P.R., D.L., C.R.M.; manuscript review, M.P.R., D.L., L.F., C.A.H., C.R.M.
Received June 3, 1998;
revision requested July 22, 1998; revision received September 9, 1998;
accepted October 7, 1998.
|
References |
|---|
|
TOPAbstractIntroductionMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| RADIOLOGY | RADIOGRAPHICS | RSNA JOURNALS ONLINE |
