HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) All Versions of this Article: 2253011335v1 225/3/719    most recent Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Williamson, K. B. Articles by Kreipke, D. L. Search for Related Content PubMed PubMed Citation Articles by Williamson, K. B. Articles by Kreipke, D. L.

Assessing Radiology Resident Reporting Skills¹

¹ From the Education and Research Institute, Department of Radiology, Indiana University, 714 N Senate Ave, Suite 100, Indianapolis, IN 46202. Received August 6, 2001; revision requested September 28; final revision received March 28, 2002; accepted June 24. Address correspondence to K.B.W. (e-mail: kenwilli@iupui.edu).

	ABSTRACT

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION REFERENCES

 
PURPOSE: To develop an Objective Structured Clinical Examination for assessing the reporting skills of radiology residents.

MATERIALS AND METHODS: The approach used in this study derives from performance-based assessment, that is, tests in which actual performance of a skill is observed and measured. For this task, 29 radiology residents and five faculty members (n = 34) independently dictated reports regarding a set of 20 cases with radiographs in 1 hour. The task was performed in a controlled environment. Data were analyzed by using analysis of variance and tests for linear trends, with the expectation of increasing performance with increasing experience.

RESULTS: Significant relationships were observed between subjects’ experience and the mean number of cases completed (F = 4.46, P = .006), the mean number of well-specified impressions (F = 5.84, P = .001), and the mean number of urgent or discrepant findings noted (F = 3.67, P = .015). Results also demonstrated a clear linear trend of increasing performance with increasing experience with each variable (P = .002, <.001, and .002, respectively, for t tests with polynomial contrasts).

CONCLUSION: The significant linear trends indicate that reporting skills increase with increasing experience in the program. This finding supports the validity of the measurement. The Objective Structured Clinical Examination provides a means of assessing radiology resident reporting skills.

© RSNA, 2002

Index terms: Diagnostic radiology, observer performance • Radiology reporting systems • Special Reports

	INTRODUCTION

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION REFERENCES

 
A recent Institute of Medicine report (1) argues that between 44,000 and 93,000 Americans die in hospitals each year as a result of preventable medical errors. A comparable number have increased hospital stays (morbidity) as well. Approximately 20% of this total (ie, morbidity and mortality considered together) die (mortality), and this is attributable to errors of diagnosis and communication. Although some researchers (2) dispute these numbers, it is clear that improving the accuracy of diagnosis—and the subsequent communication of that diagnosis to the appropriate health care provider—can substantially decrease the number of medical errors. The importance of radiologic communication is revealed by the fact that the annual number of imaging studies performed nationwide exceeds the population of the United States (3,4).

In collaboration with physicians who order the radiologic studies, radiologists play an important role in the delivery of quality health care. System compliance documents indicate the need to improve referring physicians’ orders, that is, to include sufficient patient history in the request to accurately interpret and code the images (5). Radiologists who dictate reports have a comparable responsibility to optimize the information in their reports, not only to provide information for billing but also to convey to the referring physician appropriate information that can be used for determining the diagnosis.

Those who are experienced in the field of medical education have acknowledged the problems associated with commonly used standardized examinations (6). Observational research concerning physician practice for 4 decades reveals that physicians do not typically answer complicated batteries of multiple-choice questions as a routine part of professional practice, whereas their competence evaluations consist almost entirely of such items (7). This recognition has led educators to develop performance-based assessment methods, such as clinical performance examinations with Standardized Patients and the Objective Structured Clinical Examination (OSCE), that can be used to more closely evaluate performance in actual practice (8).

Although detection skills in radiology have received considerable attention in the literature, reporting skills have not. Given the apparent need for improved communication in reporting findings of imaging studies, we developed an OSCE for assessing the reporting skills of radiology residents.

	MATERIALS AND METHODS

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION REFERENCES

 
Background Information
The task described in this study was granted an exempt review by the institutional review board at our institution. Patient informed consent was not required. The images used in this study were selected from faculty teaching files, and they had been previously read and, thus, posed no risk to patient care. Data were reported in aggregate and contained no individual identification information.

This was the first time our group had attempted to develop an OSCE, and we were quickly confronted with practical considerations. All authors discussed task parameters until a consensus was established. It was decided to impose a 1-hour time limit on the task, because this time was deemed reasonable for an experienced radiologist to report findings in 20 cases. It was further decided to limit materials to conventional radiographs.

Materials
Materials for this study were 20 cases. Each case included one or more radiographs placed in a film jacket with an accompanying requisition. In the teaching file, cases with obvious findings were chosen from among those considered by the radiologist authors (R.B.G., R.D.T., V.P.J., D.L.K.) to be typical cases of patients examined by the staff of an emergency radiology service. Requisitions were composed by one of the radiologist authors (V.P.J.). Three cases were selected with urgent findings, and discrepant requisitions for two cases were adjusted to reflect mismatched patient names or examination findings. Cases were randomized for presentation. Table 1 details the complete list of cases.

Procedures
Subjects were tested individually. Each subject was seated at a light box, and a hand-held tape recorder was provided; the set of 20 cases was presented to each subject. The subjects were told that the examination was designed to assess the reporting skills of radiology residents. Although the instructions stated that detection was not a focus of this study, participants were asked, however, to read the radiographs as well as they could and to report what they saw as if they were assigned to the service.

If subjects asked the experimenter for clarification, they were instructed to report the findings of the case as though they were assigned to service in the emergency department. If the subjects questioned the experimenter in regard to an urgent or discrepant finding, they were instructed to respond as they would if they were assigned to the service. For example, a subject said, "At this point I would call the physician. What do you want me to do with this case?" In this instance, the experimenter responded, "I want you to report in the dictation what you would do if you were assigned to the service." Two of the authors (K.B.W., J.L.S.) proctored all subjects and agreed on experimental procedures to keep the presentations as consistent as possible.

Variables and Analysis
The independent variable in this study was experience, that is, participants’ PGY or faculty status. Dependent variables included the number of cases completed by each participant in 1 hour, the number of well-specified impressions, and the number of urgent or discrepant findings noted in the findings section and reported to the referring physician.

The possible scoring range for the first dependent variable was zero to 20—the number of cases used for the assessment in this study. One point was assigned for each complete report dictated. Partial reports were assigned a score of zero. The scoring range for the second dependent variable—the impression statement—also was zero to 20. One point was assigned for each well-specified impression, and other impression statements were assigned a score of zero. A report was deemed well specified if it clearly stated a diagnosis or differential diagnosis. For example, in case 10 (Table 1), the findings were "focal lung opacity," and the differential diagnosis was "tuberculosis or pneumonia." This was contrasted with simply restating the findings (eg, "finding of focal lung opacity" with diagnosis of "infiltrates") for that same case.

Two of the coauthors—a faculty radiologist (R.B.G.) and the chief resident (T.D.W.)—determined the score by consensus. Scorers were blinded to the source of individual reports. Scoring for the third dependent variable included a maximum of eight points. One point was assigned for a correct finding with the urgent cases and another point was assigned for dictating that the referring physician was called. One point was assigned for each discrepant case if the discrepancy was noted in the dictation. A point was assigned for the notation whether or not the subject continued reading the radiograph after noting the discrepancy. A score of zero was assigned if the discrepancy was not noted.

It was expected not only that performance with respect to each of the dependent variables would differ between groups but also that performance would increase with increasing experience; that is, that performance of subjects in PGY 2 was less than that of subjects in PGY 3, that performance of subjects in PGY 3 was less than that of subjects in PGY 4, that performance of subjects in PGY 4 was less than that of subjects in PGY 5, and that performance of subjects in PGY 5 was less than that of the faculty volunteers. The basic method was a one-way analysis of variance of group means for each dependent variable. Trend tests were polynomial contrasts (9).

	RESULTS

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION REFERENCES

 
The overall F test was significant for the mean number of cases completed (P = .006), the mean number of well-specified impressions (P = .001), and the mean number of urgent or discrepant findings noted (P = .015) (Table 2). In addition, we determined a significant linear trend toward increasing performance with increasing expertise for each variable.

The number of subjects in the PGY 5 group was small, and it could be argued that inferences about that group were possibly invalid. Combining the data for PGY 5 and PGY 4 into a single group (n = 9), however, had a minimal effect on the significance level of the F test.

	DISCUSSION

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION REFERENCES

 
The trend toward increasing performance with increasing experience suggests that this OSCE is a valid measurement. Given the general observation that expertise (in this case, reporting skill) increases with experience, we would expect scores in regard to the dependent variables to increase across the grouping variable. This was, in fact, the case. The linear component proved resilient to differences in within-group variance and group size.

The strength of these data prompted our residency program directors to adopt the OSCE to be annually administered as a component of the curriculum. Informal interviews with subjects after testing confirmed the belief that residents think that they receive little or no instruction in reporting. The curriculum now includes an extended orientation for new residents that focuses specifically on reporting in five core rotations.

We have also continued to develop the OSCE with new cases and have improved scoring that includes detection as a variable. We will, however, substantially revise the testing procedure.

First, testing sessions will be conducted with five residents at a time with five identical sets of materials. We estimate that the entire resident class can be tested in less than 2 weeks rather than in several months. This will reduce the opportunity for residents to discuss the examination, a problem we encountered in this pilot study, and, thus, will limit a source of uncontrolled variance.

Second, we are increasing the number of cases in the OSCE to increase the range at the upper end of the scale. We have devised a stratified sampling method to balance case presentations across subsets of cases to reduce order bias in the materials.

Third, we also plan to implement a debriefing session for the residents after they have taken the OSCE. This will give them the opportunity to review their reports, discuss their findings, and learn from their mistakes. While the residents may not welcome the additional work, we believe that the added practice and the potential for constructive feedback will strengthen the educational experience.

This study has limitations. It has been argued that residents receive little training in reporting, because there is no consensus about what constitutes a good report (10). This lack of consensus plagues our institution as well, though the instruction resulting from this study has focused faculty attention on the issue more than ever before.

Also, the terminology used in the OSCE, specifically the word impression, has been criticized as implying a vague or imprecise conclusion (11). This term is accepted at our institution for titling report conclusions, and, thus, the OSCE reflects the environment in which the residents practice this skill. It could be argued, therefore, that the use of such terms in this study limits the generalizability of its findings to programs that also use these terms. We argue that the assessment methods represented by the OSCE are appropriate, nevertheless, and that changing report section titles to match a particular program’s usage should not affect the validity of the OSCE. This, of course, is an empirical question for which a crossinstitutional study would be needed to determine the answer.

Finally, it is recognized that results of this study can be generalized primarily to programs that rely on free-form reporting. The trend toward structured reporting and checklists eventually may cause free-form reporting to become obsolete. Until then, however, reporting will be a necessary skill for new residents, and this assessment method will help to evaluate teaching strategies targeted toward both reporting and detection.

We believe that results of this pilot study indicate an appropriate technique for evaluating resident reporting skills. Although specific psychometric properties (ie, validity and reliability) of this OSCE are currently being researched, the strength of the results of this study supports continued research in regard to the OSCE. Not only can this method be used to evaluate communications skills, an important competency designated by the Accreditation Council for Graduate Medical Education (12), but also it may well serve as an experimental measure to evaluate instructional strategies targeted toward those skills. The OSCE for assessing reporting skills of radiology residents is an important technique that should be explored further.

	FOOTNOTES

 
Abbreviations: OSCE = Objective Structured Clinical Examination, PGY = program year

Author contributions: Guarantor of integrity of entire study, K.B.W.; study concepts and design, K.B.W., V.P.J., R.D.T., D.L.K.; literature research, R.B.G., K.B.W., J.L.S.; data acquisition, J.L.S., K.B.W., T.D.W., R.B.G., D.L.K.; data analysis/interpretation, R.B.G., V.P.J., K.B.W.; statistical analysis, K.B.W.; manuscript preparation, K.B.W., R.B.G., V.P.J., J.L.S.; manuscript definition of intellectual content, K.B.W., R.B.G., V.P.J., R.D.T.; manuscript editing, all authors; manuscript revision/review, K.B.W., R.B.G., J.L.S., V.P.J., R.D.T., D.L.K.; manuscript final version approval, all authors.

	REFERENCES

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION REFERENCES

 

1. Kohn LT, Corrigan JM, Donaldson MS. To err is human: building a safer health system. Committee on Quality of Health Care in America, Institute of Medicine Washington, DC: National Academy Press, 1999.
2. McDonald CJ, Weiner M, Hui SL. Deaths due to medical errors are exaggerated in institute of medicine report. JAMA 2000; 284:93-95.[Free Full Text]
3. Sunshine JH, Bushee GR, Mallick R. U.S. radiologists’ workload in 1995–1996 and trends since 1991–1992. Radiology 1998; 208:19-24.
4. U.S. Census Bureau. Population Profile of the United States: 1997. Available at: www.census.gov/prod/3/98pubs/p23–194.pdf 2001; Accessed October 30.
5. Gunderman RB, Phillips M, Cohen MD. Improving the quality of clinical histories on radiology requisitions. Acad Radiol 2001; 8:299-303.[CrossRef][Medline]
6. Association of American Medical Colleges. Physicians for the twenty-first century: the report of the Panel on the General Professional Education of the Physician and College Preparation for Medicine Washington, DC: Association of American Medical Colleges, 1984.
7. McGaghie WC. Professional competence evaluation. Educ Res 1991; 20:3-9.
8. Reznick RK, Rajaratanam K. Performance-based assessment. In: Distlehorst LH, Dunnington GL, Folse JR, eds. Teaching and learning in medical and surgical education. Mahwah, NJ: Lawrence Erlbaum Associates, 2000; 237-243.
9. Hays WL. Statistics 3rd ed. New York, NY: Holt, Rinehart, & Winston, 1981.
10. Hall FM. Language of the radiology report: primer for residents and wayward radiologists. AJR Am J Roentgenol 2000; 175:1239-1242.[Free Full Text]
11. Orrison WW, Nord TE, Kinard RE, Juhl JH. The language of certainty: proper terminology for the ending of the radiologic report. AJR Am J Roentgenol 1985; 145:1093-1095.[Free Full Text]
12. ACGME (Accreditation Council for Graduate Medical Education) Outcomes Project. Available at: www.acgme.org/outcome/comp/compFull.asp 2001; Accessed November 30.

This article has been cited by other articles:

				K. J. Dreyer, M. K. Kalra, M. M. Maher, A. M. Hurier, B. A. Asfaw, T. Schultz, E. F. Halpern, and J. H. Thrall Application of Recently Developed Computer Algorithm for Automatic Classification of Unstructured Radiology Reports: Validation Study Radiology, February 1, 2005; 234(2): 323 - 329. [Abstract] [Full Text] [PDF]

This Article Abstract Full Text (PDF) All Versions of this Article: 2253011335v1 225/3/719    most recent Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Williamson, K. B. Articles by Kreipke, D. L. Search for Related Content PubMed PubMed Citation Articles by Williamson, K. B. Articles by Kreipke, D. L.