RADPEER quality assurance program: a multifacility study of interpretive disagreement rates¹ ☆

Article Outline

• Abstract
• Introduction
• Methods
  • Data
  • Statistical analyses
  • Modalities
  • Facilities
  • Radiologists
• Results
• Discussion
  • RADPEER and quality improvement programs
  • Comparison with results of other studies
• Conclusion
• Acknowledgements
• References
• Copyright

Abstract 

Purpose

To develop and test a radiology peer review system that adds minimally to workload, is confidential, uniform across practices, and provides useful information to meet the mandate for “evaluation of performance in practice” that is forthcoming from the American Board of Medical Specialties as one of the four elements of maintenance of certification.

Method

RADPEER has radiologists who review previous images as part of a new interpretation record their ratings of the previous interpretations on a 4-point scale. Reviewing radiologists’ ratings of 3 and 4 (disagreements in nondifficult cases) are reviewed by a peer review committee in each practice to judge whether they are misinterpretations by the original radiologists. Final ratings are sent for central data entry and analysis. A pilot test of RADPEER was conducted in 2002.

Results

Fourteen facilities participated in the pilot test, submitting a total of 20,286 cases. Disagreements in difficult cases (ratings of 2) averaged 2.9% of all cases. Committee-validated misinterpretations in nondifficult cases averaged 0.8% of all cases. There were considerable differences by modality. There were substantial differences across facilities; few of these differences were explicable by mix of modalities, facility size or type, or being early or late in the pilot test. Of 31 radiologists who interpreted over 200 cases, 2 had misinterpretation rates significantly (P < .05) above what would be expected given their individual mix of modalities and the average misinterpretation rate for each modality in their practice.

Conclusions

A substantial number of facilities participated in the pilot test, and all maintained their participation throughout the year. Data generated are useful for the peer review of individual radiologists and for showing differences by modality. RADPEER is now operational and is a good solution to the need for a peer review system with the desirable characteristics listed above.

Keywords:  RADPEER, quality assurance, observer performance, disagreement rate, interpretation, misinterpretation

Back to Article Outline

Introduction 

Quality patient care is paramount in radiology. To evaluate quality and ensure that it is high, physicians’ performance must be systematically monitored, analyzed, and improved when necessary. Peer review plays an important part in assessing physicians’ performance.

The future of medicine presumes confidence by many groups that physicians practice with skill and safety. These groups include consumers of health care, physician organizations, state medical boards, managed care organizations, and health care facilities. Interest in the competency of physicians and in measurement of such competency has increased within the health care community and the public sector over the past decade. Maintenance of competence (MOC) is a term used by many professional organizations, including the American Medical Association (AMA) and the American Board of Medical Specialties (ABMS), to describe physicians practicing with the expected level of safety and skill. Signal events in the concern about competency and the response thereto have included the development of the American Medical Accreditation Program by the AMA in 1998 and the publication of the Institute of Medicine report To Err Is Human in March 2000 [1]. An ABMS task force on the maintenance of certification, an effort parallel to MOC, which will be required by all specialty boards, produced a paper entitled: Competence Initiatives: A Status Report [2, p. 2]. In this report, Nahrwold, Youker, and Miller listed four components of maintenance of certification, and an ABMS resolution subsequently endorsed them. These components are professional standing, commitment to lifelong learning and periodic self-assessment, cognitive expertise, and the evaluation of performance in practice.

In response to the interests of the public and the health care community, the ACR convened a patient safety task force. The responsibilities of this task force included the evaluation of MOC, practice-based learning and improvement, and peer review. The task force evaluated the ACR’s continuous quality improvement (CQI) manual [3]. This manual has included sample radiology peer review programs for the past 7 years. After evaluating existing peer review programs, the task force concluded that to meet the fourth requirement of maintenance of certification, a successful peer review program must be national, uniform in structure and function across practices, accurate, facile, nonpunitive, and able to be integrated into a facility’s quality assurance program. The task force concluded that no existing programs met these criteria and that a single peer review program with objective measurements of quality was preferable to a choice of sample programs. This was the motivation for the ACR’s development of RADPEER.

RADPEER is based on the premise that whenever a new imaging study is interpreted with an older study for comparison, a peer review event assessing the accuracy of the interpretation of the older study is occurring. Therefore, if a scoring system is applied to this already existing review, and if the resulting data are tabulated, then one has a peer review system that meets the above-described goals.

In 2002, the ACR conducted a large-scale pilot test of RADPEER. This paper presents the results of that pilot test.

Back to Article Outline

Methods 

Data 

In August 2001, 20 facilities were invited to participate in the pilot test of the RADPEER peer review program. In November 2001, 14 facilities began to submit data.

Patients for whom there were related prior images of the same areas of interest on file were considered candidates for the RADPEER review process. Two of the facilities reviewed images every day of the week, approximately half the remainder reviewed images twice a week, and the rest reviewed images three times a week. (The differences in the frequency of reviews were used to examine the number of cases that would result from each option and radiologists’ willingness to participate in each.) We called the radiologists who had first interpreted the prior images the “interpreting radiologists” and the second readers the “reviewing radiologists.” The reviewing radiologists, in the course of interpreting new examinations, reviewed the interpretations of the radiologists originally interpreting the first images. These reviewing radiologists evaluated the interpreting radiologists’ original diagnoses on a 4-point scale, consisting of concurrence (rating 1) and three types of disagreement (ratings 2–4), which were entered onto optical reader cards. The categories were designated as follows: 1 = concur with interpretation, 2 = difficult diagnosis not ordinarily expected to be made, 3 = diagnosis should be made most of the time, 4 = diagnosis should be made almost every time or a misinterpretation of findings.

Any case for which the reviewing radiologist did not agree with the interpreting radiologist’s diagnosis but for which there were no clinical implications to the difference of opinion was classified as a rating of 1. Cases rated 3 or 4 were sent to a peer review committee within the practice for evaluation. The committee either validated the reviewing radiologists’ ratings of the original interpretations or rescored the original interpretations and filled out replacement cards with the new ratings. Then, all cards were sent to the ACR for data processing. (Cases rated 2 were not sent to the peer review committee because ratings of 3 and 4 were viewed as the principal actionable concern from a quality improvement standpoint.)

In principle, all cases with prior images of the areas of interest were to be entered into the RADPEER system. In practice, workload constraints would sometimes force reviewers to limit the number of RADPEER reviews to some maximum per day. To preserve randomness, the reviewers had to decide before looking at the prior images if they would review them for RADPEER, so that the images themselves could not influence whether or not reviews were performed. The reviewing radiologists were aware of the identities of the interpreting radiologists. If the radiologists who interpreted the new images were the ones who had interpreted the old images, no review cards were filled out.

The time needed for a radiologist to mark a card was estimated to be approximately 15 seconds. The optical reader cards were scanned at the ACR, and the information was thereby entered into a database. In the database, each record contained identification numbers for each of the radiologists and for the facility, the rating by the reviewing radiologist or expert committee, the modality, and the date on which the card was scanned into the database (not the date on which the procedure was performed or the date on which the image was reviewed). The identity of the radiologist associated with each identification number was known only to the facility. Thus, the ACR could not identify radiologists.

Modality was classified as plain film, ultrasound, computed tomography, magnetic resonance imaging, nuclear medicine, mammography, or interventional imaging. RADPEER also collected limited information on setting, namely, whether each facility was a community hospital, freestanding facility (none of the participants fell into this category), academic institution, or government facility. RADPEER ascertained the number of radiologists at each facility. The first cards were entered into the system on January 14, 2002. Some facilities began active participation later. This analysis uses data that were entered into the system until December 12, 2002.

Statistical analyses 

Statistical analyses of the data were performed using SAS software version 8.1 (SAS Institute Inc., Cary, NC). Statistical significance was defined as a P value of .05 or less. In regression analyses and correlations, we weighted each observation by the number of cases it involved, as is usual to adjust for differences in the variance of observations. For correlations, we report the Pearson correlation coefficients.

In the analysis, we call ratings of 2 “difficult-case disagreements” and ratings of 3 and 4 “misinterpretations.” (Recall that ratings of 3 and 4 are validated as misinterpretations by a peer review committee.)

Modalities 

For each modality, we examined if the difficult-case disagreement rate or misinterpretation rate differed significantly from the corresponding overall rate for all modalities. Such differences can arise not from actual differences among modalities but from idiosyncrasies of facilities. For example, if a facility that has particularly complex patients has an unusually high percentage of cases in a given modality, the rates for that modality will tend to be elevated. To control for this possibility, we performed a “fixed-effects” logistic regression, which recognizes that each facility may have a different overall disagreement or misinterpretation rate and investigates whether, across facilities, the rate for each modality is a consistent amount above or below that for a reference modality. (We used plain film as the reference modality because the largest number of cases consisted of plain film studies, and plain film is a general, basic form of imaging.) To illustrate the variability of modality rates, we computed the 25th and 75th percentiles of individual facilities’ rates for each modality.

Facilities 

Using ordinary least squares regression, we examined the extent to which differences among facilities in difficult-case disagreement rates or misinterpretation rates could be explained by characteristics of the facilities. In these analyses, explanatory variables employed were modality, facility type (as listed above), facility size (number of radiologists), and whether the case occurred early or late in the pilot test. This last variable was included to test whether reviewers changed their use of the rating categories over time. It was operationalized by the date each case was scanned into the database being up to April 30 (early) or from May 1 onward (late). The dependent variable was the disagreement or misinterpretation rate for each modality in each time period at each facility.

For thoroughness, we conducted an analogous logistic regression in which the dependent variable was the presence or absence of a disagreement or misinterpretation in each case.

Radiologists 

Because facilities were found to have substantial differences in disagreement and misinterpretation rates that were not explicable by any combination of the factors we considered, it would be imprudent to compare radiologists across facilities. Therefore, all comparisons of radiologists were made solely relative to other radiologists in the same facility.

We calculated the expected difficult-case disagreement (or misinterpretation) rate that each radiologist would have if the disagreement (or misinterpretation) rate by modality for that radiologist were the same as the average disagreement (or misinterpretation) rate for that modality across all radiologists in his or her facility. We then compared the observed rate for each radiologist to this expected rate to determine if the radiologist had a rate significantly higher or lower than expected. For statistical reliability, all analyses of radiologists were restricted to radiologists who interpreted at least 200 cases.

For disagreement rates by radiologist, we focused on interpreting radiologists (and only peripherally on reviewing radiologists) because the quality assurance program was primarily interested in first-reader interpretations and not in the reviewing abilities of the second readers. However, for the sake of thoroughness, we used logistic regression to test if the probability of difficult-case disagreement was different for a radiologist when he or she was a reading radiologist relative to when he or she was a reviewing radiologist.

Back to Article Outline

Results 

Fourteen facilities participated in the pilot test of RADPEER. Periodic reports, individualized to each facility and containing both statistics and explanations thereof, were produced and distributed to every participating facility; Table 1 is an example of such a report. In total, the facilities submitted 20,286 cases involving over 250 radiologists; 31 radiologists had 200 or more cases each as interpreting radiologist.

Table 1. Sample RADPEER report. Summary statistics and comparisons for each radiologist in your facility

Note. Percentages of 3s and 4s not computed for radiologists with fewer than 20 cases as data are insufficient. Probability computed only if actual number of 3s and 4s exceeds expected number and radiologist has 20 cases or more. S = significant, probability < .025; NS = not significant, probability ≥ 0.025.

Table 2 shows the misinterpretation (ratings of 3 and 4) and difficult-case disagreement (ratings of 2) rates by modality. Although the overall average rates were 0.8% and 2.9%, respectively, the table shows that approximately half the modalities had rates that differed significantly from the overall average. The logistic regressions, which controlled for differences among facilities, showed similar results, with approximately half the rates differing by a statistically significant amount from that for plain film. In general, the same rates were statistically significant, and in the same direction, in both the logistic regression and the simple comparison with the mean. The correlation of modalities’ misinterpretation rate with their difficult-case disagreement rate was 0.83 (P = .02). In 12 of 14 instances (seven modalities, each with a misinterpretation rate and difficult-case disagreement rate), the 75th percentile of facilities’ rates was approximately 3 times the 25th percentile, or even further above the 25th percentile.

Table 2. Misinterpretation rate and difficult-case disagreement rate, by modality

Table 3 shows the effects of factors other than mix of modalities on facilities’ misinterpretation and difficult-case disagreement rates. For academic institutions, both rates were approximately 1.5% higher than for community hospitals that were identical in all other characteristics considered. (Community hospitals were the reference category for facility type.) Other things equal, larger facilities had a lower misinterpretation rate than smaller ones, with the rate decreasing by 0.5% for each 10 additional radiologists, but size had no statistically significant effect on the disagreement rate. Controlling for other variables, there was no statistically significant difference in either rate between cases occurring early in the RADPEER pilot test and those that came later. In total, all the explanatory factors considered, including the mix of modalities, explained 8% of the variance in facilities’ misinterpretation rates and 23% of the variance in their difficult-case disagreement rates. Considered all together, the explanatory factors were significant at P = .001 and P < .0001, respectively. The logistic regression analyses showed similar effects for each explanatory factor and again showed that all the factors in total explained only a small part of the variation.

Table 3. Effect of factors other than modality on misinterpretation and difficult-case disagreement rates

Note. Analysis measures the variables’ effects while controlling for the effect of modalities; except for the last line, numbers are regression coefficients.

The 31 radiologists who had a minimum of 200 cases were from five different facilities. Table 4 shows that 2 of the 31 had significantly different than expected misinterpretation rates, and 3 of the 31 had significantly different than expected difficult-case disagreement rates. For the 31 radiologists, the correlation of the ratio of observed to expected misinterpretation rate with the corresponding ratio for difficult-case disagreements was .38 (P = .03).

Table 4. Misinterpretation rate and difficult-case disagreement rate, by radiologist, by facility

Note. Number of cases read by each radiologist is between 200 and 400. To preserve confidentiality, identifiers have been changed.

Logistic regression did not reveal any systematic, statistically significant effect of whether a radiologist was an interpreting radiologist or a reviewing radiologist on his or her difficult-case disagreement rate.

Back to Article Outline

Discussion 

RADPEER and quality improvement programs 

The pilot test points to a number of conclusions about RADPEER, most of them encouraging. The pilot test experience—that a substantial number of facilities participated, none had significant difficulties in carrying out the RADPEER process, and none dropped out of the pilot test—indicates the program is readily implemented.

The misinterpretation and difficult-case disagreement rates were stable over time, indicating that reviewers did not generally change their use of the rating categories in the course of the year. Misinterpretation and difficult-case disagreement rates were higher for more advanced modalities, as would be expected. The observed rates were broadly in line with those others have reported [4] (D. Soffa, personal communication, 2002), as discussed below.

The finding that only a little more than 10% of the radiologists involved achieved counts of 200 cases suggests that there may have been a problem of the nonparticipation of individual radiologists. In actuality, this disappointing result is partially explained by the fact that radiologists typically work at two to three facilities [5]; thus, the number of cases available at one facility may be only a small part of a radiologist’s total workload. An obvious conclusion is that radiology groups should apply any peer review program to all their worksites, not merely a single facility. Even allowing for this, however, poor compliance with the RADPEER program was indeed a substantial problem. Our experience is that in the current era of much increased workloads 6, 7, a serious radiologist shortage [8], and substantially decreased payments per service [9], radiologists tend to resist anything that increases their workloads or costs, even by a small amount. However, when the appraisal of quality in daily practice becomes mandated by the ABMS (or otherwise), radiologists will have to confront this conflict.

RADPEER is a less than perfect measuring system. For example, its “gold standard” is expert consensus, not verification by pathology or clinical follow-up, and reviewing radiologists may minimize their reports of disparities, knowing that the reviewed radiologists are colleagues. Moreover, as in any statistically driven peer review system, initial “positive” findings are not proof of an error in performance and should be used only as triggers for further evaluation. It must be noted that alternative quality measurement systems involve much higher time or money costs and also have potentials for distortion. For example, interpreting externally provided known cases, a system widely used in pathology, entails an added work burden equal to all the interpretations involved, and the goal of inserting the cases, unmarked, into the work stream to preclude their receiving special attention is easily suborned by a group intent on doing well on the test. Moreover, the philosophy of current quality improvement approaches (variously known as total quality management or CQI) is to employ practical, easily obtained quality measures, although they may not be altogether unimpeachable, and to concentrate resources and energy on the work of improving quality, not on measurement.

Other important lessons of the pilot test include:

Comparison with results of other studies 

Siegle et al. [4] reported a misinterpretation rate of 4.4% in a study of multiple imaging modalities in six community hospitals suspected of poor quality. These investigators used the same scale of 2 to 4 for rating case difficulty that we used. Of their 4.4% total, 1.4% was in cases rated difficult (cases rated 2), which is not far from our estimate for a difficult-case misinterpretation rate. However, the remaining 3.0% was in nondifficult cases (ratings of 3 and 4), and this is approximately 4 times our 0.8% misinterpretation rate for nondifficult cases. Perhaps a high nondifficult case misinterpretation rate is how poor performance principally manifests itself.

Soffa et al. (personal communication, 2002) reported a < 5% disagreement rate for blinded dual interpretation in a study of plain film, ultrasound, and mammography cases from a single large practice. Our estimated all-case disagreement rate for these modalities would be in the range of 3% to 3.5%.

Back to Article Outline

Conclusion 

In an era of radiologist shortages and decreased per-case reimbursement, a reasonably accurate and very-low-burden peer review system is essential to meet the need and public demand for quality. Systems requiring substantial added work or cost are unacceptable to most practicing radiologists. Indeed, RADPEER, even with its minimal additional work requirement, encountered problems in the participation of individual radiologists.

Nonetheless, RADPEER largely achieves the goal set forth in the “Introduction,” namely, to create a successful peer review program that is multipractice, uniform in structure and process across practices, nonpunitive, and able to be readily integrated in a facility’s existing quality assurance program. Because the data used in RADPEER are institutionally collected through a peer review system protected by state statute and are submitted to the ACR numerically, without identification of radiologists, confidentiality is preserved.

Subsequent steps in the maturation of RADPEER will include the collection of data from additional institutions and the evaluation of electronic data collection. The former should permit the improved analysis of differences among facilities. The latter should result in easier data submission. As of this writing (early August 2003), RADPEER has over 30 participating facilities, and more are entering the program.

Back to Article Outline

Acknowledgements 

We thank William Thorwarth, Jr., MD, for helpful ideas in developing RADPEER; E. Stephen Amis, Jr., MD, for energetically encouraging facilities to participate; and Trudie Cushing, MS, RN, RTT, for tirelessly leading the extensive and multifaceted staff work involved in developing RADPEER.

Back to Article Outline

References 

1. In:  Kohn LT,  Corrigan JM,  Donaldson MS editor. To err is human (building a safer health system). Washington, DC: Institute of Medicine, National Academy Press; 2000;
   • View In Article
2. Nahrwold DL, Youker JE, Miller SH. Competence initiatives: a status report. Report of the Task Force on Competence of the American Board of Medical Specialties
   • View In Article
3. Committee on Quality Assurance, Commission on Standards and Accreditation . A guide to continuous quality improvement in medical imaging. Reston, VA: American College of Radiology; 1996;
   • View In Article
4. Siegle RL, Baram EM, Reuter SR, Clarke EA, Lancaster JL, McMahan CA. Rates of disagreement in imaging interpretation in a group of community hospitals. Acad Radiol. 1998;5:148–154
   • View In Article
   • Abstract
   • Full-Text PDF (1323 KB)
   • CrossRef
5. Deitch CH, Chan WC, Sunshine JH, Shaffer KA. Profile of U.S. radiologists at middecade (overview of findings from the 1995 survey of radiologists). Radiology. 1997;202:69–77
   • View In Article
   • MEDLINE
6. Bhargavan M, Sunshine JH. Workload of radiologists in the United States in 1998–1999 and trends since 1995–1996. Am J Roentgenol. 2002;179:1123–1128
   • View In Article
7. Sunshine JH, Bushee GR, Mallick R. U.S. radiologists’ workload in 1995–1996 and trends since 1991–1992. Radiology. 1998;208:19–24
   • View In Article
   • MEDLINE
8. Bhargavan M, Sunshine JH, Schepps B. Too few radiologists?. Am J Roentgenol. 2002;178:1075–1082
   • View In Article
9. Hogan C, Sunshine JH. Financial ratios in diagnostic radiology practices: their variability and trends. Radiology. In press
   • View In Article

• 1 RADPEER is a quality assurance product of the ACR. For information or to implement RADPEER, contact the ACR’s Quality and Safety Department at 1 (800) 227-5463, extension 4384.