HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Figures Only Full Text (PDF) Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Itagaki, M. W. Articles by Pile-Spellman, J. Search for Related Content PubMed PubMed Citation Articles by Itagaki, M. W. Articles by Pile-Spellman, J.

Factors Associated with Academic Radiology Research Productivity¹

¹ From the University of Illinois College of Medicine, Urbana, Ill (M.W.I.); and Department of Radiology, Columbia University, College of Physicians & Surgeons, New York, NY (J.P.). Received August 31, 2004; revision requested November 8; revision received December 15; accepted January 18, 2005. Address correspondence to M.W.I., Department of Medicine, Winthrop University Hospital, 222 Station Plaza North, Suite 509, Mineola, NY 11501 (e-mail: mitagaki{at}gmail.com).

	ABSTRACT

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION References

 
PURPOSE: To determine factors associated with research productivity among all university radiology departments in the United States.

MATERIALS AND METHODS: As an observational study, this was exempt from institutional review board approval. All 47 299 radiology articles from 1996 through 2003 that were indexed in the National Library of Medicine MEDLINE database were studied. Each article was assigned a "publication impact" score based on the impact factor of its source journal. These citations were then matched, along with National Institutes of Health (NIH) extramural grant, residency, fellowship, faculty, and geographic data, to 109 individual medical school radiology departments. Raw citation count was used to measure research quantity, and aggregate publication impact was used to measure quality. Regression analyses were used to compare the relationship between the study variables and research quality and quantity measures on the departmental and individual faculty level.

RESULTS: Finalized statistical models accounted for 75%–88% of variance in productivity. NIH funding had a significant and positive association with all measures of research productivity (P < .001), with one article associated with $167 980 in funding and one publication impact unit associated with $83 271 in funding. Large resident program sizes (P < .001) and the presence of fellows (P = .007) also had a significant association. Geographic region, salary, and faculty characteristics had no detectable association. Extrapolations based on these results estimated the cost of annual global radiology research at $907 million, with the U.S. component at at least $417 million. NIH funding accounted for 45% of the U.S. component.

CONCLUSION: NIH funding, resident program size, and fellow characteristics are significantly associated with academic research output.

© RSNA, 2005

	INTRODUCTION

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION References

 
Maintaining and enhancing research in academic radiology is a critical prerequisite for the continued development of new and more useful clinical imaging modalities. This goal was the focus of a recent collaborative conference involving the Academy of Radiology Research, the American College of Radiology, the American Roentgen Ray Society, and the Radiological Society of North America (1). Government recognition of the importance of radiology research has increased, as demonstrated by the establishment of the National Institute of Biomedical Imaging and Bioengineering in 2000 and the steady increase in National Institutes of Health (NIH) radiology funding to U.S. medical schools from $105 million in 1996 to $276 million in 2003 (2).

Despite the clear importance of research productivity in radiology, it is unclear exactly what factors are associated with it. A traditional measure of academic success has been the amount of NIH research funding obtained by a department. Other measures include the number of articles or books published. Because there is no single generally accepted measure of academic research productivity, the purpose of this study was to determine factors associated with research productivity among all university radiology departments in the United States.

	MATERIALS AND METHODS

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION References

 
Publications
This was an observational study of existing publicly available data; hence, it qualified for an exemption from institutional review board approval. The unit of study was the medical school radiology department. Peer-reviewed radiology department publications were obtained by searching the National Library of Medicine PubMed database for all MEDLINE articles that contained the word radiology in the citation affiliated institution field. Articles published from 1996 through 2003 were included. Each article that was returned by the search engine was then cross matched with the journal impact factor of the given journal to obtain the publication impact of that article. We used the 2002 journal impact factor values from the Thompson ISI Journal Citation Reports (3). Journal International Standard Serial Numbers were used to make the association. If an article did not have an associated journal impact factor, it was assigned a value of zero. Articles were assigned to medical schools only if the medical school was explicitly identified in the article affiliation field. This exclusionary protocol was used because many hospitals employ both medical school faculty and nonfaculty physicians and it is impossible to determine academic association purely on the basis of the hospital affiliation. This protocol prevents nonfaculty publications from being counted incorrectly toward a medical school but tends to undercount legitimate articles.

Although the publication impact factor variable accounts for the relative prestige of the journal in which an article is published, it does not take into account the type of article published. It is generally accepted that, all other things being equal, case reports typically have lower scientific value than do reports of clinical trials. To determine whether well-funded departments produced a different mix of article types relative to poorly funded departments, we ranked medical school departments according to NIH funding and then compared them with the number of articles of a specific type they produced. The types of articles we examined were case reports, review articles, and reports of clinical trials of all types.

Residency and Fellowship Programs
Data for Accreditation Council for Graduate Medical Education (ACGME)–approved residencies in diagnostic radiology and radiology fellowships were obtained from the 2003 Graduate Medical Education Database of the American Medical Association (4). The information was categorized according to program and included resident, fellow, and faculty statistics, as well as medical school affiliations and degree of affiliation as reported by the program directors. Programs are often affiliated with multiple medical schools and medical schools with multiple programs, but a program typically has only one major medical school affiliation. If a program director declared more than one major affiliation, information obtained directly from the ACGME Web site (5) was used to assign the program to a medical school. In two instances, the number of teaching faculty members for a program was missing and was obtained directly from the program Web site. In instances in which a medical school had major associations with multiple residency or fellowship programs, the research output and study variables of all the affiliated programs were aggregated and counted toward that medical school.

NIH Grants
NIH extramural grants for medical school radiology departments from 1996 to 2003 were obtained from the NIH Web site (2). All grants were inflation adjusted to 2003 dollars by using the online Consumer Price Index Inflation Calculator tool (6). Additional funding variables were created by using standard mathematic transformations. Medical schools that received more than $5 million in annual NIH funding were "clustered away" from the rest. A variable, "highly funded by NIH," was created to identify these highly funded schools.

Geographic Regions and Salaries
Schools were assigned to geographic regions by using the nine standard region divisions (7) used by the American Medical Association and the U.S. government. Regional university radiology faculty salary data were obtained from the Association of American Medical Colleges (8). We hypothesized that high faculty salaries encourage physicians to remain in academics and high private practice salaries encourage faculty members to leave. Definitive radiologist salary data for states or regions were unobtainable. Therefore, we used average physician (all specialties) salary data according to region from the American Medical Association (7) as a proxy for regional radiologist salaries and assumed that regions that have high general physician salaries tend to have high radiologist salaries.

Statistical Analysis
The variables studied are summarized in Figure 1. We examined four measures of research output. Research volume was measured by using the annual number of articles published during the study period, with no consideration given to the type of article or the journal in which the article was published. Research quality was measured by using publication impact, a variable calculated by assigning each article a score equal to the impact factor of the source journal. Research volume and quality were measured on both departmental and individual faculty levels. Statistical analyses were performed by using statistical software (SPSS for Windows, version 11.5; SPSS, Chicago, Ill). The statistical test used was multiple linear regression. All residual values were examined to check for nonuniformity. Because of the large number of variables studied, the threshold for significance was set at P .01. All data collection and analysis was performed by one author (M.W.I.). The statistical tests were chosen and the design of the analysis was performed by both authors (M.W.I. and J.P.) in consensus.

View larger version (93K): [in this window] [in a new window] [Download PPT slide]   Figure 1. Regression analysis study variables.

	RESULTS

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION References

View larger version (25K): [in this window] [in a new window] [Download PPT slide]   Figure 2. Graph shows estimates of the cost of articles and publication impact units on the departmental and individual levels, with 95% confidence intervals (error bars). These estimates were derived with the four statistical models, with the mean estimate representing the average of the departmental and individual faculty estimates for the given productivity measure.

The highly funded by NIH variable was also significant (P = .001). Receiving more than $5 million in annual NIH funding was associated with a boost in research quality but not volume. Departments that crossed this funding threshold enjoyed an additional 50 publication impact units annually, or an additional 0.88 impact units on an individual faculty level.

The number of fellows was a significant (P = .007) factor only with regard to individual faculty volume. The relationship was complex and nonlinear. Overall, the number of fellows had a negative coefficient, but the coefficient for the ratio of fellows to faculty was positive. The manner in which fellows interacted with faculty thus depended on the faculty size. In a department with 40 faculty members and 15 fellows, each fellow was associated with an increase of 0.32 article per year per faculty member. In a department with 100 faculty members, each fellow was associated with a decrease of 0.09 article per year. The inflection point occurred at about 75 faculty members. The average department in this study had approximately 55 faculty members and three fellows. In such a department, fellows would be expected to contribute 0.02696 citation per year per faculty member, or 1.48 citations total for 55 faculty members. This amounts to 0.49 citation per fellow per year. Because many departments have only a handful of fellows, the net effect was rather small.

NIH Funding and Citation Type
Poorly funded departments were more likely than well-funded departments to publish case reports than reports of clinical trials and accounted for a larger fraction of all published case reports than all published reports of clinical trials (Fig 3). Nonetheless, well-funded departments still accounted for an overwhelming majority of all articles of all types. Departments in the top quartile of NIH funding accounted for 47% of all case reports, 64% of all reviews, 67% of all clinical trials, and 63% of all articles overall.

View larger version (27K): [in this window] [in a new window] [Download PPT slide]   Figure 3. Graph shows cumulative percentages of total articles, case reports, review articles, and reports of clinical trials from medical school radiology departments according to departmental NIH funding rank. Well-funded departments accounted for far more articles than poorly funded departments.

	DISCUSSION

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION References

 
Departmental Volume
The combination of departmental NIH funding and residency program size accounted for 86.5% of all variation in research volume on the departmental level. The significance of program size observed in the present study was in contrast to evidence that radiology residents have a negative effect on clinical productivity (9). In the present study, medical schools with residency programs with many ACGME-approved residency positions gained one article annually for every 2.6 resident positions. Surprisingly, the number of affiliated residency programs was not found to be significant; neither was required research rotation or research mentoring for residents.

In the present study, no faculty variables were significantly related to departmental research volume. It seems reasonable to expect that having many faculty members would result in more articles or that the presence of faculty researchers with PhDs or part-time nonphysician research assistants might be important. However, this analysis did not reveal that faculty size, sex, degree, and full-time status were statistically significant factors. Our results contradict the findings of one study (10), which indicated that large faculties were positively associated with more publications in the field of emergency medicine, although this may be attributable to structural differences in research between the two specialties.

Departmental Quality
By using productivity impact, the prestige and approximate quality of the journals in which articles appeared were taken into account in our analysis. Journal impact factors have been criticized as not accurately measuring the quality of any single article (11). It is true that a case report in a given journal may not have the same importance as a report of a milestone clinical trial in the same journal. However, journal impact factors have been found to agree with clinician beliefs about high-quality journals (12). The impact factor of the source journal is also the single most important factor in predicting the number of citations that a given article will subsequently receive (13).

As was the case with the departmental volume model, NIH funding and residency program size were found to be significant positive indicators of productivity. An additional variable, highly funded by NIH, was also significant. This suggests a critical mass effect: High levels of departmental funding may produce an extra boost in research quality but not quantity. As with the departmental volume model, faculty characteristics, number of fellows, geographic region, and salary were not significant.

Individual Faculty Volume
Strong similarities existed between the departmental and individual faculty volume models. Annual per-faculty NIH funding was the funding variable that proved most significant for individual faculty volume. The associated funding per citation of $180 602 differed from the departmental estimate of $155 357 by only 14%. As in the departmental model, having large residency programs contributed positively to output in the individual faculty volume model. Interestingly, however, this was the only model in which fellows had a significant role.

Individual Faculty Quality
The factors associated with individual faculty quality were basically the same as those associated with departmental quality: per-faculty NIH funding, number of resident positions per program, and highly funded by NIH status. There was good agreement between the coefficients of the departmental and individual faculty quality models. One publication impact unit was associated with $90 984 in NIH funding. None of the faculty, fellow, regional, or salary variables were found to be significant.

Comparisons between Models
Certain variables had a strong association with all measures of research productivity on both the departmental and individual scales. NIH funding was a highly significant variable in all four analyses. It is unsurprising that funding level has an impact on productivity because funded faculty members are typically academic veterans with a proved track record of success. Having financial independence allows a great deal of organizational freedom and flexibility that nonfunded faculty members do not enjoy. Furthermore, it is possible that the funding effect may compound—that is, that funding results in publications that in turn result in more funding.

Large resident program sizes also had a significant and positive effect on research output. It may be that residents provide a pool of willing and skilled labor that can be utilized by faculty researchers. Furthermore, the 4-year length of the diagnostic residency program provides sufficient time for residents to become invested in research projects. Interestingly, resident research mentoring and required research rotations were not significant predictors of research output. One hypothesis that may account for this finding is that programs with strong resident research support also tend to have more NIH funding and that the funding rather than the resident research is the more powerful predictor of research output. The role of residents was in contrast to the role of fellows, who were found to positively affect only individual research volume and only in smaller departments. The short length of fellowship programs and the transient nature of fellows may provide an explanation.

None of the faculty variables were found to be statistically significant in any of the models. Faculty size, educational degree, sex, and employment status apparently had no effect on research output; alternatively, the effect of these variables could have been accounted for by another variable such as NIH funding. Results of other studies (14) have shown that faculty characteristics—in particular, demands for high clinical volume—have a deleterious effect on academic productivity. The effect of salaries in both academia and competing private practice was not significant. Evidence exists in the emergency medicine literature (10) that geographic region is a factor in research output, with departments in the western United States tending to be more productive. Our analysis indicated that in radiology, geographic region was not a significant determinant.

Valuing Radiology Research
The results of the statistical models yielded four estimates of the cost of radiology research output. There was strong agreement between the departmental and individual cost estimates of radiology articles and publication impact. These figures allowed estimates of global, U.S., and NIH-funded research productivity to be made. These figures likely represent lower limits of the true expenditure, because our method did not capture radiology research published by nonradiology departments such as engineering and physics. The difference between the amount of direct NIH funding to U.S. departments and the value of the actual research output of those departments is likely accounted for by faculty salary; non-NIH sources of funding; and the unfunded labor of faculty members, residents, and students. It should be noted that our protocol tended to underassign publications to medical school departments; hence, these figures tend to underestimate the output of all U.S. departments. Still, the proportion of global clinical radiology research output that is attributable to the NIH is considerable.

Study Limitations
Measuring the true quality of a scientific publication is an inherently subjective enterprise. Although we have endeavored to measure both volume and quality on the individual and departmental level, both of these parameters are still approximations of true productivity and are not perfect. Sometimes only the affiliation of the first author is mentioned in a citation. This was a limiting factor in this study. Authors and departments that are consistently listed after the first author would therefore be undercounted. The MEDLINE citations were limited to those that contained the word radiology in the affiliated institution field. Although this search term captures a majority of clinical radiology departments, radiology research published by other departments, such as engineering, was not counted. Publications were assigned to medical schools with exclusionary criteria that required an explicit statement of affiliation; hence, the protocol tended to undercount citations. In instances in which a medical school had multiple affiliated training programs, the output and faculty of the extra training programs were counted toward the medical school. This captured the research productivity of the affiliated programs but also tended to dilute the individual faculty productivity measures in the main program.

The NIH includes grants for radiation oncology in its reporting of radiology grant data. Prior to 2002, the NIH reported grants to radiology departments as "Radiology" grants. Since 2002, grants have been reported as "Radiation-Diagnostic/Oncology" grants. Owing to the huge overlap between these specialties with respect to grant reporting, research focus, and departmental organization, it was not possible to separate them. However, as long as publicly reported NIH figures for "Radiology-Diagnostic/Oncology" are used, the predictive value of statistical models should not be unduly diminished. NIH grant figures represent total inflation-adjusted grant awards, and no attempt was made to analyze the direct or indirect subcomponents.

No attempt was made to analyze trends over the 8-year study period. Data from all 8 years were aggregated, and any time-sensitive information was lost. The geographic granularity of the salary data was less than ideal. The salary data were reported in aggregate by region. Local market effects of salary differences within a city or locality thus could not be captured. Furthermore, the private practice comparison salary data were for all physicians, not just radiologists. Comparison salary data thus did not account for variations in compensation that radiologists may experience relative to other physicians in a given region. These shortcomings, while substantial, were applied equally to all departments in this study, thus mitigating their effects.

Conclusions
Research productivity in academic radiology can be measured on both the departmental and individual levels by using publication volume and quality. This analysis encompassed all academic U.S. radiology department research output from 1996 through 2003. NIH funding, residency program size, and, to a lesser extent, the number of fellows and the ratio of fellows to faculty, accounted for between 75% and 88% of all variation in research output between departments, depending on the productivity measure used. One published article was associated with $168 980 in funding, and one publication impact unit was associated with $83 271 in funding. Medical schools with larger residency programs had greater research output, with each additional resident accounting for about one-third of a publication per year. The number of fellows had a small but significant effect on the number of articles individual faculty members published. Faculty characteristics, geographic region, and salary had no effect.

The worldwide value of annual clinical radiology research was $907 million, with the U.S. component being at least $417 million. NIH-funded medical schools accounted for 97.5% of the total U.S. publication impact, and it is estimated that NIH funding directly accounts for 45% of all U.S. clinical radiology research output. The importance of continued government support for academic research in radiology cannot be overemphasized.

	ACKNOWLEDGMENTS

 
The authors thank Brian Lee, BA, for his assistance; J. David Liss, MFA, and Columbia University for financial support; and Emerson Lane, BA, for his constructive comments.

	FOOTNOTES

 

Abbreviations: ACGME = Accreditation Council for Graduate Medical Education • NIH = National Institutes of Health

Authors stated no financial relationship to disclose.

Author contributions: Guarantors of integrity of entire study, M.W.I., J.P,; study concepts/study design or data acquisition or data analysis/interpretation, M.W.I., J.P.; manuscript drafting or manuscript revision for important intellectual content, M.W.I., J.P.; approval of final version of submitted manuscript, M.W.I., J.P.; literature research, M.W.I.; statistical analysis, M.W.I., J.P.; and manuscript editing, M.W.I., J.P.

	References

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION References

 

1. Alderson PO, Bresolin LB, Becker GJ, et al. Enhancing research in academic radiology departments: recommendations of the 2003 consensus conference. Radiology 2004;232:405–408.[Abstract/Free Full Text]
2. National Institutes of Health historical grant data. http://grants1.nih.gov/grants/award/awardtr.htm. Accessed August 25, 2004.
3. Thompson ISI journal citation reports. http://www.isinet.com/products/evaltools/jcr/. Accessed June 13, 2004.
4. Brotherton SE, Rockey PH, Etzel SI. US graduate medical education, 2004-2005: trends in primary care specialties. JAMA 2005;294:1075–1082.[Abstract/Free Full Text]
5. Accreditation Council for Graduate Medical Education Web site. http://www.acgme.org. Accessed June 13, 2004.
6. U.S. Department of Labor Bureau of Labor Statistics Web site. http://www.bls.gov/. Accessed August 25, 2004.
7. American Medical Association. Physician Socioeconomic Statistics 2003 edition. Chicago, Ill: American Medical Association, 2003.
8. Association of American Medical Colleges. Report on Medical School Faculty Salaries, 2000–2001. Washington, DC: Association of American Medical Colleges, 2002.
9. Cohen MD, Hawes DR, Hutchins GD, McPhee WD, LeMasters MB, Fallon RP. Activity-based cost analysis: a method of analyzing the financial and operating performance of academic radiology departments. Radiology 2000;215:708–716.[Abstract/Free Full Text]
10. Henderson SO, Brestky P. Predictors of academic productivity in emergency medicine. Acad Emerg Med 2003;10:1009–1011.[CrossRef][Medline]
11. Walter G, Bloch S, Hunt G, Fisher K. Counting on citations: a flawed way to measure quality. Med J Aust 2003;178:280–281.[Medline]
12. Saha S, Saint S, Christakis DA. Impact factor: a valid measure of journal quality? J Med Libr Assoc 2003;91:42–46.[Medline]
13. Callaham M, Wears RL, Weber E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA 2002;287:2847–2850.[Abstract/Free Full Text]
14. Eschelman DJ, Sullivan DL, Parker L, Levin DC. The relationship of clinical and academic productivity in a university hospital radiology department. AJR Am J Roentgenol 2000;174:27–31.[Abstract/Free Full Text]

This article has been cited by other articles:

				G. P. Krestin Evaluating the Quality of Radiology Research: What Are the Rules of the Game? Radiology, November 1, 2008; 249(2): 418 - 424. [Full Text] [PDF]

				M. W. Itagaki Impact of the National Institutes of Health on Radiology Research Radiology, April 1, 2008; 247(1): 213 - 219. [Abstract] [Full Text] [PDF]

				L. Arrive and A. Miguel-Dasit Declining Number of Publications by American Authors in Radiology Radiology, January 1, 2008; 246(1): 330 - 331. [Full Text] [PDF]

				A. S. Brody Computed Tomography Scanning in Cystic Fibrosis Research Trials: Practical Lessons from Three Clinical Trials in the United States Proceedings of the ATS, August 1, 2007; 4(4): 350 - 354. [Abstract] [Full Text] [PDF]

This Article Abstract Figures Only Full Text (PDF) Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Itagaki, M. W. Articles by Pile-Spellman, J. Search for Related Content PubMed PubMed Citation Articles by Itagaki, M. W. Articles by Pile-Spellman, J.