In case anyone had any doubt what the purpose of this year's exam was:
You're in a small program? We don't care about you and want to see you fail. You don't belong in this field. It's absurd that failures are punished by being forced to retake the exams alongside the clinical, increasing the chances of failing all 3. Why not try and help those who failed pass rather than making it even more difficult for them the next time? Wonder how things would be different if it were the large programs that had most of the failures? Isn't it funny how they published a biased opinion about small programs before this year's exam was even administered? What a coincidence.
TO: Association of Residents in Radiation Oncology (ARRO) Executive Committee American College of Radiation Oncology (ACRO) Resident Committee FROM: Valerie P. Jackson, MD, Executive Director Brent Wagner, MD, President
DATE: October 5, 2018
SUBJECT: Response to September 26, 2018 letter of concern
We appreciate the thoughtful comments you expressed in your cover note and open letter dated September 26, 2018. We understand the frustration from a small group of candidates who did not perform as well as intended on one or both of the recent ABR radiation oncology basic science qualifying exams. Below are responses to those concerns, as well as a few observations from the ABR perspective. You indicate that angry residents have reached out to you anonymously and non-anonymously. We have also been the recipient of similar communications, many of which demonstrate a significant misunderstanding of the ABR exam development, standard-setting and scoring processes, the fundamentals of various organizational responsibilities, and the essence of the process by which the determination of the knowledge and skill set expected of trainees in radiation oncology is made. ABR volunteer leaders will be available to provide a detailed response to your communication at the upcoming ARRO and ADROP meetings in San Antonio. We believe that some general observations and clarification of a number of misconceptions are appropriate at this time. Regarding ABR transparency with regard to its processes, policies, and results – we make concerted effort to share information that is useful to both candidates and programs while taking appropriate actions to safeguard the exam content. Pass rates for all exams are routinely provided to department chairs and program directors, and have been posted online. In 2016-2017, a change in web-based exam results reporting was established for what was thought to be an improvement in understanding. The ABR recognizes that for initial certification (IC), aggregate reporting may be less informative. Thus, we will be returning to our previous practice of annual posting. Candidates are provided with quartile scores, rather than raw scores, because quartile positions more readily permit assessing the performance of the individual in comparison to the peer group. The logistics of the criterion-referenced standard-setting method (Angoff) have been widely described in ABR publications and in a host of academic peer-reviewed journals and texts. The Angoff method is employed by the majority of American Board of Medical Specialties (ABMS) member boards and is considered a best-practice for this type of professional assessment instrument. The Angoff method has been found to be highly reliable, reproducible, and valid. The ABR has tracked the validity of its own use of the Angoff standard-setting system and has never had deviations from discriminatory norms. The statistical analysis that is performed tracks year-over-year performance by all candidates, which is helpful from a historical perspective and assists department chairs and program directors in assessing their programs and trainees. While informative, this analysis does not take into account factors related to individual exam questions, or the many variables associated with variation in candidates, training programs, and importantly, addition of new material and deletion of outdated material as clinical care and basic science advance. The rigorous and routine ABR psychometric analyses focus on reliability, difficulty, and discriminatory accuracy of individual questions, and in that regard, the performance of the exams this year was well within metric reliability. As we have indicated previously, the exam development and implementation process have remained essentially unchanged for many years, including development by many of the same individuals. After the exam is administered, each candidate’s exam response data is reviewed to ensure that his or her data is complete and accurately recorded. The number of responses is confirmed as the correct number for the exam that the candidate took. An initial scoring is completed, and all scores are reviewed. Then, each question on each exam is reviewed statistically. Any question that does not perform as expected is sent to the appropriate committee for review. The committee determines whether the keyed answer is truly correct and that there are not other provided answer options that could be confusing. If the committee decides that the keyed answer is incorrect, or confusing, they may remove the question from the exam and the scoring process. Scores are then recalculated and checked again for accuracy before posting to myABR. Your open letter also references a lack of change in didactic education, available study materials or in-service exam scores. The ABR has no direct means of evaluating quantity or quality of didactic education, as this is the role of the Accreditation Council for Graduate Medical Education (ACGME). However, careful analysis of performance by program size, which will be presented in greater detail at the upcoming meetings, suggests a direct relationship between program size (as one possible surrogate for didactic education) and exam performance. In both the physics and radiation and cancer biology exams, candidates training in programs of 6 or fewer candidates had a remarkable difference in pass/fail rates when compared to their peers who trained in larger programs. These differences were further magnified by the fact that 61% (55 of 90) of the programs reviewed had 6 or fewer trainees, and in the current exams, 46% of the peer group (100 of 217) were trained in those small programs. A majority of candidates who failed a basic science exam failed both exams, including a significant number of candidates who had failed the exam(s) previously. These findings raise concern regarding exam preparation. With regard to the performance on inservice exams, there is no valid basis to compare performance in those assessment tools as compared to the ABR exams, which are developed by different people, for different purposes, and, in comparison to the ABR exams, are subjected to no psychometric controls, validation or review. The radiation oncology study guides provided by the ABR are developed to offer guidance only as to topics which might be included in exams. A recent review of those documents indicates that these guides provide that information. The basic sciences in radiation oncology represent dynamic domains, with constant addition of new material. These items are included in the study guide and, as such, it is incumbent upon residency training programs to prepare trainees for these new ideas, terms and concepts. Comparison to materials provided for our diagnostic radiology (DR) colleagues is not appropriate; the domains assessed in DR exams include dozens of imaging modalities, hundreds of normal and pathologic entities, and thousands of imaging variations, with a primary assessment of a bi-modal correct or incorrect diagnostic decision. The provision of greater detail on distribution of potential material was essential because of the enormity of potential material and the introduction of an entirely new DR core (qualifying) examination several years ago. The basic distribution of radiation oncology exam material has generally followed the previously published tri-annual clinical practice analysis (CPA) survey. The CPA has directly informed exam development in such specific ways as a reduction in pediatrics and brachytherapy content, based on declines in those practices by radiation oncologists in the field. Your letter also refers to “standard” texts which have been basic resources for radiation oncology trainees for generations. Regrettably, a significant number of active cancer scientists agree that those texts are outdated. The ABR is committed to working with our volunteers to provide more updated reference sources for trainees and educators. We agree that a lack of specialty-wide, consensus-driven curricula in physics and radiation and cancer biology is problematic, leading to remarkably heterogeneous teaching and preparation. However, curriculum development is outside the scope of the ABR’s mission: this activity is more appropriately managed by the ACGME Radiation Oncology Review Committee (RO RC) and various stakeholder specialty organizations. We have encouraged those stakeholders to update the previously developed physics curriculum, and to develop for the first time, a radiation and cancer biology standardized curriculum. Curriculum development should be associated with a greater attempt to provide homogeneous levels of basic science education to trainees. In conclusion, the ABR stands by the reliability and supportability of its exams. We will continue to work with chairs, program directors, basic science educators and stakeholder organizations to better prepare candidates for the certification process.
