Outdated Test Questions - What to Do?

[Johnny Sunshine]

---

Members don't see this ad.

We all know medicine is a continually evolving field. And even when a change in the standard of care occurs, we tend to be forgiving for a little while during the transition time, when some people may hang on to the old ways for a bit.

But when it starts to interfere in testing, such as the RISE and various board exams, I get really ticked. 😡

The new recommendation is to call eosinophilic esophagitis if you see just 15 eos per HPF instead of 20? Okay, fine. But if you have a test question on that a full year after this is publicly announced, don't ding my answer as wrong if I call EE when there are 17 eos in the image.

We've moved to a full, standardized Bethesda system on Paps, and calling adenocarcinoma in situ from a Pap is no longer deemed appropriate? Fine, but then don't have adenocarcinoma in situ as the correct answer to a test question.

The proper follow-up for LSIL is colposcopy? Great. But why have that as an "official" stance and as a test question if the majority of clinicians actually just do repeat Pap in 6-12 months?

Writing letters or emails to those who have a monopoly on writing the exams is useless. I've even spoken face-to-face to some of the AP board organizers at a national conference on this issue, and was not encouraged by their replies (basically, they admitted they are out of the loop and so it may take up to 5 years for current, proper practice answers to make it into the exams).

So what are we to do? Learn both the real and outdated info for everything, and just hope we remember which data to use on a test versus which to use in real life?

Grr. Not likely to find an answer here, but I think it's worth chewing the fat over.

 

[dermpathdoc]

We all know medicine is a continually evolving field. And even when a change in the standard of care occurs, we tend to be forgiving for a little while during the transition time, when some people may hang on to the old ways for a bit.

But when it starts to interfere in testing, such as the RISE and various board exams, I get really ticked. 😡

The new recommendation is to call eosinophilic esophagitis if you see just 15 eos per HPF instead of 20? Okay, fine. But if you have a test question on that a full year after this is publicly announced, don't ding my answer as wrong if I call EE when there are 17 eos in the image.

We've moved to a full, standardized Bethesda system on Paps, and calling adenocarcinoma in situ from a Pap is no longer deemed appropriate? Fine, but then don't have adenocarcinoma in situ as the correct answer to a test question.

The proper follow-up for LSIL is colposcopy? Great. But why have that as an "official" stance and as a test question if the majority of clinicians actually just do repeat Pap in 6-12 months?

Writing letters or emails to those who have a monopoly on writing the exams is useless. I've even spoken face-to-face to some of the AP board organizers at a national conference on this issue, and was not encouraged by their replies (basically, they admitted they are out of the loop and so it may take up to 5 years for current, proper practice answers to make it into the exams).

So what are we to do? Learn both the real and outdated info for everything, and just hope we remember which data to use on a test versus which to use in real life?

Grr. Not likely to find an answer here, but I think it's worth chewing the fat over.

Isn't there an option to make a comment on the question? Certainly for USMLE and COMLEX all questions that have comments on them concerning current validity will be reviewed by a panel concerning standard of care and if the validity is found to be not current, then the question will be removed from the graded items pool. Certainly a number of practice board exams do the same. I would seriously worry if they don't because test psychometricians take this extremely seriously.

 

[yaah]

Well for RISE you can give specific feedback. It should have an impact, although it is likely to impact your score.

For boards I don't know - I suspect you could complain and if you're right they will likely throw the question out. I really don't understand why it would take 5 years to have an impact though.

 

[pathstudent]

Worrying and complaining about test questions is very weenieish and med schoolish. If you know your stuff so well that you know youre forced to choose between a formerly correct answer and a current practice correct answer, then you will have no problem passing the boards. And that's what is all about.

Let go of the weenie med student mentality.

 

[yaah]

Well, the truth is that sometimes questions are wrong and this fact isn't caught before it is actually used. It is unlikely, however, that someone who notices an incorrect question and flags it is going to fail the exam because of this one question. Someone who notices an incorrect question is highly likely to have a high score anyway because they are SMRT.

 

[mlw03]

Darn it - I wrote a long reply and website fritzed out.

Basically I'd said I feel the poster's frustrations, and don't think it's a weenie med student mentality. This is a national licensing exam, and a very expensive one at that. The ABP should, in my view, ensure questions reflect standards of practice at the time they are administered. That means diagnostic terminology, ancillary testing, etc. Stuff that is controversial shouldn't be tested. But I do not think it's unreasonable to expect this of the ABP.

 

[pathstudent]

Darn it - I wrote a long reply and website fritzed out.

Basically I'd said I feel the poster's frustrations, and don't think it's a weenie med student mentality. This is a national licensing exam, and a very expensive one at that. The ABP should, in my view, ensure questions reflect standards of practice at the time they are administered. That means diagnostic terminology, ancillary testing, etc. Stuff that is controversial shouldn't be tested. But I do not think it's unreasonable to expect this of the ABP.

Disagree. Even if you have a point. It is a waste of time and I would bet that for virtually all questions the answers are current and correct. It is best to just move on with your life.

 

[pathololology]

Disagree. Even if you have a point. It is a waste of time and I would bet that for virtually all questions the answers are current and correct. It is best to just move on with your life.

I totally disagree with this. If they are getting paid to write these questions and charging us >$2000 to take these tests in addition to flying halfway around the world and paying for an overpriced hotel room to take part in their monopolized business, then we have a right to protest bad questions that have an effect on our licensing. Take it from recent test taker. . . not all questions and answers are correct (they even through 1 out right in the middle of the test). BTW that general mentality is not a very good one and is not "weeneish". Allowing people to walk all over you and shrugging your shoulders and saying "oh well . . . what can I do they are probably right" is an easy way to get more of the same for the remainder of your life. . .

 

[pathstudent]

I totally disagree with this. If they are getting paid to write these questions and charging us >$2000 to take these tests in addition to flying halfway around the world and paying for an overpriced hotel room to take part in their monopolized business, then we have a right to protest bad questions that have an effect on our licensing. Take it from recent test taker. . . not all questions and answers are correct (they even through 1 out right in the middle of the test). BTW that general mentality is not a very good one and is not "weeneish". Allowing people to walk all over you and shrugging your shoulders and saying "oh well . . . what can I do they are probably right" is an easy way to get more of the same for the remainder of your life. . .

Well you don't know what you missed or didn't miss, so it is not really an issue. So you can't claim that they has a "wrong" answer as the right answer.

 

[pathololology]

Well you don't know what you missed or didn't miss, so it is not really an issue. So you can't claim that they has a "wrong" answer as the right answer.

That's true but that's not the point and not what the OP was stating. The issue is when you see 2 right answers or read answers that doesn't make sense with the information given (like a blood bank panel without the correct answer as a choice which happened a few years ago). If everyone just says "oh well the correct answer was probably there and we are just too dumb to realize it" rather than pointing it out to the test committee, then the test committee may never realize it.

 

[TMZ2007]

A couple people I know who just took and passed the Cyto boards both complained that there were "next step in management" questions based on the old ASCCP guidelines. They left vociferous comments after the exam, but who knows what good that'll do...

 

[Johnny Sunshine]

A couple people I know who just took and passed the Cyto boards both complained that there were "next step in management" questions based on the old ASCCP guidelines. They left vociferous comments after the exam, but who knows what good that'll do...

I just got into a discussion/debate with an attending over this exact thing; ASCCP guidelines for management. Thankfully, my review books seem to be up to date. 🙂

Honestly, there will always be test questions that discuss a topic that has recently come under scrutiny or for which the standard of care has recently changed. In such cases, I suppose the best thing that can be done is to inform the proctors that the question may not be appropriate for inclusion in that year's exam (depending on the answer options given).

Unfortunately, you won't necessarily know you've stumbled upon such a question unless you are aware of both the old and new guidelines. Suddenly, you find yourself in a situation where you basically have to know more about a topic than do those who have written the exam. That's a bit disconcerting. But I don't know if there's anything to really be done. Perhaps it is just one of those facts of life you can only shrug and deal with.

 

[deleted314957]

I'm quite sure that the ABP really doesn't care about this issue. They got your money and they now have you continually enslaved with MOC.

 

[mlw03]

I'm quite sure that the ABP really doesn't care about this issue. They got your money and they now have you continually enslaved with MOC.

Sage advice from the "old-timer" - I cannot imagine anyone in the ABP actually gives a poop about a few test questions each year being out of date. When I talk to Canadians about their exam, it sounds like a much better determinator of one's ability to actually practice pathology competently.

 

[pathololology]

At the AP/CP boards they actually ask you to alert them to any issues. It may not always be a correct answer issue, but instead something may have been left out of the question that was meant to be put in there or that they thought they included in there. My main point to everyone is that it doesn't hurt to question things and sometimes you will be right. You can't always have the "we'll they are probably smarter" attitude because that may not even be the issue. It could have been a glitch or something else unintentionaI, and if nobody alerts them to it, then a bad question will be graded and will affect scores. The "they are smarter so why should I bother" attitude is a piss-poor passive attitude that leads to complacency and is the bane of progress in anything whether it be politics, society, ethics, academics or otherwise.

My friends who took the derm boards had stated that over the years many questionable questions had been removed since enough people complained about them and the questions have become much more straightforward. It pisses me off when people make fun of people who question test questions or test answers. We are forking out $1,000's of dollars to an entity that we have no power over and quite honestly have no alternative too. They could charge us $50,000 for AP/CP boards and we would all complain about it, but in the end we would have to pay it b/c there is no other option and people won't hire if you aren't board certified. We will still fly halfway across the continent to wake up at 5am West Coast time and take a very long 2 day exam completely jet-lagged. The least we can do is challenge an unfair question or wrong answers. Don't get me wrong, I don't believe I am in anyway smarter than anyone designing these questions, but I bet there are some people taking these tests that actually are.

I just got into a discussion/debate with an attending over this exact thing; ASCCP guidelines for management. Thankfully, my review books seem to be up to date. 🙂

Honestly, there will always be test questions that discuss a topic that has recently come under scrutiny or for which the standard of care has recently changed. In such cases, I suppose the best thing that can be done is to inform the proctors that the question may not be appropriate for inclusion in that year's exam (depending on the answer options given).

Unfortunately, you won't necessarily know you've stumbled upon such a question unless you are aware of both the old and new guidelines. Suddenly, you find yourself in a situation where you basically have to know more about a topic than do those who have written the exam. That's a bit disconcerting. But I don't know if there's anything to really be done. Perhaps it is just one of those facts of life you can only shrug and deal with.

 

[oldfatman]

At the AP/CP boards they actually ask you to alert them to any issues. It may not always be a correct answer issue, but instead something may have been left out of the question that was meant to be put in there or that they thought they included in there. My main point to everyone is that it doesn't hurt to question things and sometimes you will be right. You can't always have the "we'll they are probably smarter" attitude because that may not even be the issue. It could have been a glitch or something else unintentionaI, and if nobody alerts them to it, then a bad question will be graded and will affect scores. The "they are smarter so why should I bother" attitude is a piss-poor passive attitude that leads to complacency and is the bane of progress in anything whether it be politics, society, ethics, academics or otherwise.

My friends who took the derm boards had stated that over the years many questionable questions had been removed since enough people complained about them and the questions have become much more straightforward. It pisses me off when people make fun of people who question test questions or test answers. We are forking out $1,000's of dollars to an entity that we have no power over and quite honestly have no alternative too. They could charge us $50,000 for AP/CP boards and we would all complain about it, but in the end we would have to pay it b/c there is no other option and people won't hire if you aren't board certified. We will still fly halfway across the continent to wake up at 5am West Coast time and take a very long 2 day exam completely jet-lagged. The least we can do is challenge an unfair question or wrong answers. Don't get me wrong, I don't believe I am in anyway smarter than anyone designing these questions, but I bet there are some people taking these tests that actually are.

Maybe you should sue the board and insist on a quality assurance audit

 

[pathololology]

Maybe you should sue the board and insist on a quality assurance audit

Good one fatman. I'll let you waste your dollars if you want. The more feasible way would be to create a 2nd board (like the North American Board of Pathology or something similar) and offering all tests online or all tests that can be given at any Sylvan learning center (not dissimilar to Steps 1, 2, and 3) in peoples home state or city and charge them less money to take the test. The ABP would literally be out of business overnight. You wouldn't have to fly anywhere, wouldn't have to get a hotel, and it would be cheaper. It wouldn't be much different from lab accreditation that can be done by CAP, JCAHO, AABB, or COLA. Why shouldn't there be more than one certifying board for physicians?

 

[deleted314957]

If they were not sanctioned by the American Board of Medical Specialties that entity would have no standing/clout whatsoever. Look at such non-starters such as the American Board of Cosmetic Surgery and it's couple dozen kin.

 

[longlakeboy]

Hey, you can't expect them to update their questions. After all, it costs $5000 to write each question on the exam. 😀