| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Computer Applications |
1 From the Radiology Department, Middlemore Hospital, Hospital Rd, Otahuhu, Auckland 6, New Zealand. Received August 18, 2004; revision requested October 21; revision received December 17; accepted January 20, 2005. Address correspondence to A.J.D. (e-mail: Adoyle{at}middlemore.co.nz).
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
MATERIALS AND METHODS: This study was conducted according to the principles of the Declaration of Helsinki (2002 version) of the World Medical Association. The institutional clinical board approved the study; informed consent was not required. Seven observers independently assessed randomized anonymous digital radiographs of the wrist from 259 subjects; 146 had fractures, and 113 were healthy control subjects (151 male and 108 female subjects; average age, 33 years). Follow-up radiographs and/or computed tomographic scans were used as the reference standard for patients with fractures, and follow-up radiographs and/or clinical history data were used as the reference standard for controls. The PC was a standard hospital machine with a 17-inch (43-cm) color monitor with which Web browser display software was used. The PACS workstation had two portrait 21-inch (53-cm) monochrome monitors that displayed 2300 lines. The observers assigned scores to the radiographs on a scale of 1 (no fracture) to 5 (definite fracture). Receiver operating characteristic (ROC) curves, sensitivity, specificity, and accuracy were compared.
RESULTS: The areas under the ROC curves were almost identical for the PC and workstation (0.910 vs 0.918, respectively; difference, 0.008; 95% confidence interval: –0.029, 0.013). The average sensitivity with the PC was almost identical to that with the workstation (85% vs 84%, respectively), as was the average specificity (82% vs 81%, respectively). The average accuracy (83%) was the same for both.
CONCLUSION: The results of this study showed that there was no difference in accuracy of observer performance for detection of wrist fractures with a PC compared with that with a PACS workstation.
© RSNA, 2005
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
Clinicians frequently have access only to PCs in the clinical work area. Furthermore, radiologists are often called on to look at radiographs on a PC when a workstation is not available (eg, at home or in an office, ward, or clinic). There are a number of PC-based picture archiving and communication systems (PACS) in existence, and they are described in the radiology literature (1,2). Therefore, it is important to know how accurate the PC-based interpretation of radiographs is in comparison to that with a PACS workstation (the standard of reference). There has been very little investigation of this issue in general and almost none related to fractures (3–7).
Wrist injuries are common and, in our institution and in others that we know of, wrist radiographs may be first viewed with a PC by clinicians and/or radiologists. Wrist fractures are often subtle, thus providing a reasonably rigorous test of observer performance. Our null hypothesis was that, in the detection of wrist fractures on radiographs, there would be no statistically significant difference in observer performance between that achieved with a standard hospital PC and that achieved with the best-quality PACS workstation available in our institution.
Thus, the purpose of our study was to retrospectively compare the accuracy of observer performance in the detection of wrist fractures on computed radiographs with a standard PC compared with a dedicated PACS workstation.
|
MATERIALS AND METHODS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
Subject Selection and Reference Standard
The study subjects were selected by searching the PACS (Pathspeed, version 8.1; GE Medical Systems, Milwaukee, Wis) for consecutive radiographs of the wrist obtained because of acute trauma. Clinically, the subjects were suspected of having fractures and were seen from March 2001 to May 2002. Subject and radiograph selection was performed by a musculoskeletal radiologist with 15 years of experience (A.J.D.) and a 3rd-year medical student (J.L.F.). They worked together in consensus, and neither acted as an observer in the study.
Subjects in whom initial radiographs were not obtained at our institution, those in whom standard views were not obtained, and those who did not have a clear history of acute trauma or who did not have adequate radiographic and/or clinical follow-up to confirm or exclude the presence of a fracture were excluded. Only subjects with acute fractures or those with no fracture were included in the study. Patients with fractures in which the acuteness was questionable were excluded. Patients with arthritic changes were included unless the changes were so severe as to make interpretation difficult.
Apart from the study group, the same two authors selected 21 additional subjects to form a pilot group with which to train the observers before they started the study proper. This group consisted of 11 patients with fractures (of whom five had two fractures, for a total of 16 fractures) and 10 healthy control subjects.
The study group of 259 subjects included 113 control subjects who did not have fractures and 146 patients who had one or two fractures. Fifty-three patients had two fractures; thus, there were 199 fractures. The average age of subjects was 33 years (range, 10 months to 96 years). There were 151 male and 108 female subjects. Fractures were confirmed or excluded by checking follow-up radiographs and/or computed tomographic scans and by checking patient records for further medical treatment to establish the reference standard. This process was performed by the senior author (A.J.D.), a musculoskeletal radiologist with 15 years of musculoskeletal radiology experience.
Of the 146 patients with fractures, 72 were deemed to have subtle fractures and 74 were deemed to have more obvious fractures. This determination was made in consensus by the same two authors who selected the subjects. A fracture was determined to be subtle if it fulfilled one of the following criteria: It had less than 1 mm of displacement and less than 10° of angulation or it was not in the radius or scaphoid bone and was near the boundary of the field of view. A typical example of a subtle fracture is shown in the Figure.
|
|
Radiographs of structures were obtained as follows: wrist, posteroanterior and lateral views in 154 patients; scaphoid bone, which consisted of a posteroanterior view of the wrist with ulnar deviation, an oblique 15° view with ulnar deviation and straight ray, an oblique 15° view with ulnar deviation and 20° beam angulation toward the elbow, and a lateral view of the scaphoid bone in 62 patients; forearm and wrist, posteroanterior and lateral views in 37 patients; wrist, bilateral views in four patients; hand, posteroanterior and lateral views in one patient; and hand, oblique view in one patient.
Image Acquisition and Retrieval
All radiographs were obtained by using photostimulable phosphor plate radiography. Plates (Agfa-Gevaert, Mortsel, Belgium) were 24 x 30 cm and were read in a compact digitizer (Pathspeed MP3010; GE Medical Systems) with a matrix of 2048 x 2577 pixels. The resulting digital images were postprocessed at a quality assurance workstation (Pathspeed VIPS; GE Medical Systems) with proprietary software (Mimosa 1.1.06; Agfa-Gevaert). These 12-bit images were sent to the PACS short-term storage device (a redundant hard-disk array) and to the optical disk archive (Pathspeed, version 8.1; GE Medical Systems). From the short-term storage, they were retrieved at a commercially available workstation as 12-bit images and at a PC by means of a commercially available Web browser (Pathspeed Web, version 8.1; GE Medical Systems). With this Web browser system, only those portions of the image that can be displayed at the chosen window and magnification levels are retrieved at the PC. The entire image information, however, is available to the PC user, provided the window and magnification tools are used appropriately.
PACS Workstation
The workstation (Pathspeed 2A; GE Medical Systems) consisted of a computer (Pentium II; Intel, Santa Clara, Calif) that was running at 400 MHz, with 512 MB synchronous dynamic RAM. The operating system (Windows NT Workstation, version 4.0; Microsoft, Redmond, Wash) ran proprietary software (Pathspeed, version 8.1; GE Medical Systems) linked to two high-brightness gray-scale portrait monitors (Barco Medical Imaging Systems, Kortrijk, Belgium) that were 21 inches (53 cm) with a display of 1728 x 2304 lines, 146 dots per inch, and 8 bits. The display was calibrated and controlled by using proprietary software (MediCal Pro, version 2.02.04; Barco Medical Imaging Systems). As measured with a narrow-angle luminance meter (Minolta LS-100; Konica Minolta, Tokyo, Japan) with a display of a Society of Motion Picture and Television Engineers test pattern, these monitors had a maximum luminance of 311 and 312 candelas (cd) per square meter.
PC-based System
The PC (HP Vectra VL 420; Hewlett Packard, Palo Alto, Calif) contained a central processor (Pentium 4; Intel) that was running at 1.6 GHz, with 256 MB of synchronous dynamic RAM and a standard video card (Rage 128Pro II GL AGP4X/PCI, driver version 4.3.3226, 4.0.0; ATI Technologies, Markham, Ontario, Canada), with 16 MB of synchronous dynamic RAM with an internal digital-to-analog converter at 350 MHz.
The operating system (Microsoft Windows NT Workstation, version 4.0; Microsoft) was used with proprietary image review software (Pathspeed Web, version 8.1; GE Medical Systems) linked to a 17-inch (43 cm) color monitor (HP72; Hewlett Packard), with a display of 1024 x 768 lines and 8 bits and running at 65 536 colors with a 75-Hz refresh rate. As measured in the same fashion as the workstation, this monitor had a maximum luminance of 122 cd/m2.
The brightness and contrast controls on the PC were set by using a Society of Motion Picture and Television Engineers test pattern to make the contrast steps as even as possible and then were fixed for the duration of the study. Gamma correction is not possible with this operating system.
Image Analysis
Before the observers started the study, they underwent training with both the PC and the workstation by using the pilot group of 21 subjects.
For the actual study, all observers were blinded to subject details. The subject information (including all other procedures performed and radiologist reports) was removed, and the subjects were assigned an arbitrary number from 1 to 259 for the purpose of this study. The subjects were then classified into four blocks of 50 and one block of 59. The observer viewed the radiographs from the first 50 subjects with the workstation, switched to the PC to view the radiographs from the next 50 subjects, and viewed the radiographs from the next 50 subjects with the workstation; they continued in this manner until all radiographs obtained in the 259 patients had been viewed.
After an interval of between 4 and 8 weeks, the observers viewed the radiographs again. At this time, each block was randomized internally in terms of number order. The viewing platform was switched so that the observer viewed the radiographs from the first 50 subjects with the PC, those from the next 50 subjects with the workstation, and so on, until the last block of radiographs from 59 patients had been viewed and the observations were recorded.
The observers had no time limit for their observations. They had access to and were shown how to use all the tools appropriate for image manipulation. With both systems, the observers were encouraged to magnify the images to use the maximum spatial resolution the monitors were capable of displaying. With the PC, magnification of the entire image with subsequent panning was recommended, whereas with the workstation either this method or use of the magnification glass was suggested. These functions were essentially instantaneous with both the PC and the workstation. With both the PC and the workstation, the default hanging protocol was one on one.
The workstation and PC were housed in similar environments and remained in place for the duration of the study. Ambient light was reduced to that coming from the monitors and from a doorway to an interior corridor, which was approximately 10 feet away from the workstation and approximately 10 feet away from the PC. With use of a lux meter (DSE Q-1400; Dick Smith Electronics, Sydney, Australia), the background illumination readings were less than 5 lux at each location.
Scoring System
The observer assessed whether fractures were visible on each patient's radiograph and then assigned scores to the radiograph on a scale of 1–5 as follows: score 1, definitely normal with no fracture present; score 2, probably normal with no fracture present; score 3, unsure; score 4, probable fracture; and score 5, definite fracture. If the radiograph was assigned a score of 4 or 5, the observer recorded which bone or bones they thought were involved with fractures. The observers were told that some subjects had no fractures, some had one fracture, and others had more than one fracture. At the end of the study, the observers were asked to give subjective comments about which viewing platform they preferred, how often they used image magnification, and whether they took longer to complete the study with the PC or with the workstation.
Statistical Analysis
Receiver operating characteristic (ROC) curves were generated and analysis of variance (8) was performed by using standard software downloaded from the University of Chicago Web site (LABMRMC 1.0B Beta, version 3, 1997; Charles E Metz, Department of Radiology, University of Chicago, Chicago, Ill). Results for total variance and variance were generated according to viewing platform, subject, observer, platform and subject, platform and observer, observer and subject, and platform, observer, and subject.
An estimate of the power of the study was determined on the basis of the 95% confidence intervals for the area under the ROC curves (Az). It was assumed that the Az values followed a Gaussian distribution with a standard error that was 0.25 times the 95% confidence interval. This enabled calculation of the probability of the demonstration of a given value of the difference between the Az values at a given significance level. It also enabled calculation of the difference in Az that could be demonstrated with a given probability at a given significance level. For a power of 80% at the 5% significance level, the detectable difference is 2.8 times the standard error. We assumed that a detectable Az difference of 0.05 (just over 5%) or less would be an appropriate target for such a clinical study, given that we would expect an Az of approximately 0.9 for the workstation.
For calculation of sensitivity and specificity, a score of at least 3 was determined to be positive for a fracture, and a score of 1 or 2 was determined to be negative for a fracture. This should best reflect clinical reality, in which a doubtful result would usually lead to performance of follow-up radiography on the basis of the assumption that a fracture may be present. Sensitivity was calculated as the proportion of positive scores recorded in the group with fractures, and specificity was calculated as the proportion of negative scores recorded in the control group. These calculations were repeated, with a score of 4 or 5 determined to be a positive finding and a score of 3 or less determined to be a negative finding.
Accuracy was calculated as the sum of true-positive and true-negative results divided by the total number of patients. To estimate the degree of uncertainty experienced by the observers, the number of unsure scores of 3 was counted. The number of patients in whom the observer correctly identified a fracture but ascribed it to the wrong bone was also counted. The 
2 test was used to compare these rates.
To estimate the presence of a memory effect, the number of subjects was counted in whom the observer determined an incorrect diagnosis and then changed it between the evaluation with the PC and that with the workstation. The observers were also asked whether they could remember the individual patient's radiographs between rounds of observations.
|
RESULTS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
ROC, Analysis of Variance, and Power Analysis
Results of ROC analysis showed no significant difference in observer performance between the PC and the workstation. The Az for the PC was 0.9103 (standard error, 0.0194), and that for the workstation was 0.9181 (standard error, 0.0219). The difference was –0.0078 (standard error, 0.0084), and the 95% confidence interval was –0.0285 to 0.0129. The Az values and 95% confidence intervals for the individual readers showed no significant difference between the PC and workstation (Table).
|
The different viewing platforms (PC or workstation) did not contribute to variance (F ratio = 0.8555).
The magnitude of the 95% confidence interval for the Az is 0.0414, which yields a standard error of 0.01. Therefore, the power calculation shows that the study has a greater than 99% chance of revealing an Az difference of 0.05 at a significance level of 5%. It has an 80% chance of revealing an Az difference of 0.028 at the 5% significance level.
Score of 3: Positive or Negative
For the group of seven observers, the average sensitivity with the PC when scores of 3 or more were determined to be positive for fracture was almost identical to that with the workstation (85% [124 of 146] vs 84% [123 of 146]). The average specificity was also almost identical (82% [93 of 113] for the PC vs 81% [92 of 113] for the workstation). The average accuracy (83% [215 of 259]) was the same for the PC and the workstation. These values were also very similar for each observer.
When these values were recalculated with scores of 3 or less as negative findings, the results were almost identical. The sensitivity decreased two percentage points for both the PC and the workstation to 83% (121 of 146) and 82% (120 of 146), respectively. The specificity increased four points for the PC and five points for the workstation to 86% (97 of 113) each. The accuracy increased by two points for the PC to 85% (220 of 259) and by one point for the workstation to 84% (218 of 259).
The total number of subjects in whom the observers identified a fracture but ascribed it to the wrong bone was quite low. For the PC, findings were wrong bone–positive in 22 (2.5%) of 868 true-positive observations; for the workstation, findings were wrong bone–positive in 16 (1.8%) of 861 true-positive observations. With the 2 test, no significant difference was found.
The total number of scores of 3, or unsure, returned by the seven observers was 50 with the PC and 55 with the workstation. With the 2 test, the difference was not significant.
Memory Effect
When scores of 3 or more were used as positive findings, there were 610 incorrect diagnoses. In 219 (36%) of these incorrect diagnoses, the observer changed the category when the observer switched between the evaluation with the PC and that with the workstation. The observers reported that there were very few radiographs that they could remember seeing before while they performed the second round of observations.
|
DISCUSSION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
With such a result, the possibility of a type II error (failure to reject the null hypothesis because of the lack of statistical power) must be considered. The fairly narrow 95% confidence intervals for the Az scores and the calculated 99% chance of detection of an Az difference of 0.05 at the 5% significance level, however, make it unlikely that a type II error has occurred. As expected, statistically significant differences were demonstrated between the observers and between the patients (those with fractures vs controls).
Another possible flaw that could lead to an artifactual lack of difference between the scores with the PC and the workstation is that of memory effect. The nature of the study design, however, makes it very unlikely that there is any substantial memory effect. Radiographs from the five subsets of subjects were viewed in a different order with the PC and with the workstation. An interval of several weeks was allowed between the reading of each case with the two platforms, and the order of cases within the subsets was randomly altered between the two sets of readings. The fact that in just more than one-third of the incorrect diagnoses there was a change in category between readings with the PC and those with the workstation further supports the absence of any memory effect.
We conclude that, provided each viewing platform is used to maximum effect, our observers were just as accurate at detection of wrist fractures with a PC as they were at detection of these fractures with a workstation. In consideration of the subtle nature of many of the fractures included in this study, we may expect a similar result for other more obvious fractures. Our observers had a wide range of experience and expertise and, although the overall performance varied with experience, the equivalence of the PC and workstation did not. Extrapolation to other skeletal radiographs may be reasonable, especially in light of findings of a previous study in which the researchers demonstrated that there was no monitor-related difference in the assessment of digital radiographs for hyperparathyroidism (9). Generalization to other organ systems causes more problems and necessitates further investigation, as will be discussed later.
One implication of these findings is that, in the case of acute wrist fractures, both radiologists and clinicians can make a primary diagnosis by reviewing the radiograph with a PC without having to find a workstation to confirm their findings. This may effectively increase severalfold the number of devices from which a primary diagnosis of fracture may be determined in most institutions. The financial burden of providing adequate facilities for radiographic review in many health care enterprises potentially can be decreased.
In our study, the ambient light was kept at low levels for both PC and workstation viewing. We believe that control of ambient light is important for accurate interpretation, especially with low-luminance displays. Decreased observer performance with increasing ambient light has been previously shown to affect color monitors more than it does bright monochrome monitors (6). Therefore, both radiologists and clinicians who seek to interpret digital radiographs with PCs should ensure that ambient light is kept low.
We do not conclude that a PC can replace a workstation for routine diagnostic reporting purposes, especially where substantial volumes of work are involved. Although we did not conduct a formal time-and-motion study, the observers said they took longer to complete image review with the PC. This was related partly to the absence of work list functionality with the PC, partly to slower image retrieval and manipulation, and perhaps partly to the longer searching and decision-making times that have been shown to be associated with diminished display luminance (10).
Results of previous small studies have shown no difference in observer performance with a PC compared with that with a workstation for reading digital radiographs obtained in patients with rheumatoid arthritis (7) and in a set of varied digitized screen-film radiographs obtained in the emergency department (3). Results of another study in which a PC was used to examine digitized screen-film chest radiographs in patients with lung cancer (4) showed no statistically significant difference in observer performance compared with examination of the original radiographs. Those studies involved limited numbers of patients and thus had a lower statistical power than did this study.
The equivalence of performance with the PC and the workstation may not hold true for the viewing of radiographs of other organ systems. Results of some studies (11,12) have shown differences in observer performance for detection of subtle pulmonary nodules and for that of abnormalities on mammograms viewed with monitors having differing screen phosphors. Findings in another study (13) showed decreased performance with low luminance for pneumothorax and rib fracture. These studies, however, were conducted with digitized screen-film radiographs, and the radiologist observers were not allowed to adjust window width or window level settings. Because commercial digital radiographic systems have sophisticated postprocessing programs that optimize image contrast, it is possible that similar studies performed with primary digital radiographs would have different results.
Similar considerations apply to another study in which researchers demonstrated differences in accuracy of fracture interpretation on digitized screen-film radiographs, depending on the monitor system used (5). In that study, a PC was used; however, only the lower resolution of two digitizers and a lossy compression system were used. The cases were read sequentially from the lowest to the highest resolution system followed by the original screen-film radiographs, which introduced a learning effect. The authors indicated that the PC system used was cumbersome and that some observers admitted defeat with it for some of the cases.
Researchers in two other studies have shown differing observer performance with digital radiographs that was related to monitor resolution and luminance. In one of these studies (6), a PC produced performance that was inferior to that of a workstation for review of a phantom. The observers in that study, however, did not necessarily use the image magnification and/or windowing tools. Furthermore, it can be difficult to translate the results of a phantom study to clinical image review. Those authors concluded that the PC graphic card was to blame for much of the difference in performance. We used a standard graphic card in our study, however, and showed no difference in performance.
In the other study (14), printed images of digital radiographs were compared with those displayed with a monitor. Again, a phantom with simulated pulmonary lesions was used. This study did not involve the use of a PC. The authors found that, although differences were present, "[w]ith use of systemic magnification, performance with all monitors was equal to that with hard copies." This is precisely what we found for fracture detection.
We deliberately encouraged the observers to use the full range of image manipulation tools available and, in particular, to magnify the image with both the PC and the workstation so that the spatial resolution with the monitor screen would match or exceed that of the primary digital image wherever possible. Even with the workstation, there would otherwise be a reduction in the modulation transfer function of the image because the primary digital image would contain more pixels than would the monitor display area. It is our opinion that use of image tools in this manner should be a routine part of soft-copy reporting. Without magnification, some of the subtle fractures in this study were not visible with either platform. Previous work with digitized screen-film radiographs has also shown that display resolution is important in the detection of scaphoid bone fractures (15).
In summary, we showed that there is no difference in the accuracy of observer performance in the detection of wrist fractures on digital radiographs with a standard PC compared with that with a high-quality workstation. In our opinion, important factors in achievement of this result are the use of image magnification to employ the maximum spatial resolution of the original digital image and the use of an environment with low ambient light.
|
ACKNOWLEDGMENTS |
|---|
|
FOOTNOTES |
|---|
Abbreviations: Az = area under the ROC curve • PACS = picture archiving and communication system • PC = personal computer • ROC = receiver operating characteristic
Authors stated no financial relationship to disclose.
Author contributions: Guarantor of integrity of entire study, A.J.D.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; literature research, A.J.D.; clinical studies, A.J.D.; statistical analysis, A.J.D., J.L.F.; and manuscript editing, all authors
|
References |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  D S Brettle Display considerations for hospital-wide viewing of soft copy images Br. J. Radiol., July 1, 2007; 80(955): 503 - 507. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 B. F. Branstetter IV Basics of Imaging Informatics: Part 1 Radiology, June 1, 2007; 243(3): 656 - 667. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M H Brem, C Bohner, A Brenning, K Gelse, T Radkow, M Blanke, P M Schlechtweg, G Neumann, I Y Wu, W Bautz, et al. Evaluation of low-cost computer monitors for the detection of cervical spine injuries in the emergency room: an observer confidence-based study. Emerg. Med. J., November 1, 2006; 23(11): 850 - 853. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| RADIOLOGY | RADIOGRAPHICS | RSNA JOURNALS ONLINE |
