Since you are one of the writers, I'm curious - why the desire to test numbers so hard and also rare cancers? For example, the question that asks for the recurrences on the CW and regional LNs with no RT in NSABP B-18 to B-27 (the ranges of percents goes from 1.5 - 7% for CW, and 1.5 - 4.5% for regional LNs) or the most common site of LN metastasis in a patient with penile cancer or the primary treatment of male urethral carcinoma or the staging of ampulla of Vater cancer. The incidence of these questions on the exam are about a 1000x higher than the incidence of the disease 🙂
Have to laugh about this...would you believe that EVERY SINGLE YEAR I walk into the room where the clinical group is vetting questions, the first thing out of my mouth is "Is it
really necessary to test on penile, urethral and ampulla of Vater cancers? The residents hate us for that, you know."
But seriously, what it comes down to is what the teams of question writers choose to write about...and you can't really
demand they write on certain topics but not others, de-emphasize the percentages, or mix things up more, when they're all volunteers in the first place. (Personally, I do try to ask politely, which sometimes works but sometimes doesn't.)
That's also why it seems like practically the same questions or at least, question content, is repeated year after year.
With the key, would it be possible for the ACR to also list the percent of examinees that got a particular question correct? That might be helpful, as well - i.e. what I think is trivia might actually be common knowledge and something I need to know.
Unfortunately, all the stats concerning how well each question "tests" (i.e., does the distribution of responses clearly discriminate between the lower, mid and upper third of test performances) are kept confidential by the testing company.
I've always been interested in this myself actually, if for no other reason than to get a sense of what kinds of questions work and what don't.
That being said, one thing I do know is that when you look at the answer key and you see "NS" next to a particular question, that means "Not Scored"...in other words, that the question didn't "discriminate" well and that either nearly everybody either got it right or wrong.
Lousy test was obviously written on a plane, while coming back from some nice conference.
I should be so lucky as to be even attending some nice conference, let alone find the time to be writing questions on a plane home!
😉
The reality is that I spend quite a bit of time each summer on writing, editing, fact-checking, composing answer justifications (with references) and trying to balance question content for both my own questions and those of my fellow writers. And THEN I have to work with the testing company to get the things shortened to contain the minimum number of distracting adjectives, and worded the way they want, while still trying to retain clarity and relevance. And THEN, in the fall, I have to go and defend my questions
again in front of the assembled panel of group leaders from all the biology, clinical and physics specialties and sub-specialties.
Granted, I do get to stay at a very cushy Ritz Carlton for a couple of nights (
😎 )...but otherwise, the entire process is actually a labor of love, not to mention hard work.