Is USMLE an adaptive CAT test?

[xnihil]

Let's face it - USMLE would not really be computerized had they not intended to make use of computer-based test techniques. So the question I am asking is: is the USMLE a computerized adaptive test (CAT) such as that used in the NCLEX-RN, despite the fact that full-scale implementation of CAT on the USMLE has not yet been formally announced? CAT assesses the examinee's knowledge base by using statistical theory to adjust the content and difficulty of each question based on prior answers and overall exam performance.

CBT also offers the potential for new question formats. With multiple-choice questions, for example, the test-taker may be able to arrive at the correct answers by unintentional or subliminal hints offered by the answer choices themselves. Such cueing can make current exams less accurate tools for assessment. Cueing effects might be eliminated, and test accuracy improved, by the use of computerized long-menu questions. The test-taker with a long list of possible answers rather than 4 or 5 choices. Based on their studies, they suggest scores from tests using long-menu questions are more comparable to exams with open-ended questions rather than multiple-choice exams. As you may be aware of, NBME is already trying to reduce cueing - in the hope of improving examinee assessment - by using clinical and laboratory simulations, multimedia presentations, and "point and click" question formats.

---

Members don't see this ad.

 

[lord_jeebus]

No, I remember that the NBME specifically denied this a few years ago. I don't think this has changed.

 

[Rabbit Hole]

CAT assesses the examinee's knowledge base by using statistical theory to adjust the content and difficulty of each question based on prior answers and overall exam performance.

I really don't think the USMLE is a CAT. The exam allows you to skip through questions and change answers so it doesn't make sense that the content/difficulty of each question is based on prior answers. Also, I'm not quite sure how they could account for the scattered experimental questions when computing the score (unless all those questions are in one block..?).

CBT also offers the potential for new question formats. With multiple-choice questions, for example, the test-taker may be able to arrive at the correct answers by unintentional or subliminal hints offered by the answer choices themselves.

They do give a few hints in the exam.. right after you lock the first question in a sequence item and can't go back to change it :cry:.

Questions that appear on the USMLE have to adhere to very strict guidelines and are evaluated on more than one level before they make the cut. NBME publication: Constructing Written Test Questions For the Basic and Clinical Sciences

 

[Rabbit Hole]

I forgot to add that the good people at NBME & ECFMG are currently reviewing all steps of the USMLE.. so who knows what the exam will look like or how it will be administered in the future. Guinea pig for life.

 

[ifu]

I really don't think the USMLE is a CAT. The exam allows you to skip through questions and change answers so it doesn't make sense that the content/difficulty of each question is based on prior answers.

It's called CAST (Computer-Adaptive Sequential Testing), I believe. The basic test design model for CAST is a multistage test, with adaptive testing capabilities at the level of subtests or testlets (blocks).

 

[ifu]

Also, I'm not quite sure how they could account for the scattered experimental questions when computing the score (unless all those questions are in one block..?).

Kaplan maintains that anywhere from 30 to 35 questions on the exam are included so that they can be pretested and evaluated for use in other exams. In addition, as many as 10 to 15 other questions may be eleminated from the scored pool after the exam results are reviewed.

 

[chaperone]

Assuming that USMLE is indeed an adaptive (CAST or whatever the thing is called) test - does this mean that we really have to focus on the difficult items, since they weigh more than the easy ones? Or is it that we have to focus on blocks, instead of individual items, trying to get as many as possible right within a block? Also, does this mean that the first blocks (probably the first 4) are the ones that really matter, that the last 3 will have little impact on your score, since the computer would have found your ability level by then? Does this mean that the test tries to determine at which level you answer 60% correctly and 40% incorrectly, accounting for people guessing answers correctly?

 

[Raryn]

Assuming that USMLE is indeed an adaptive (CAST or whatever the thing is called) test - does this mean that we really have to focus on the difficult items, since they weigh more than the easy ones? Or is it that we have to focus on blocks, instead of individual items, trying to get as many as possible right within a block? Also, does this mean that the first blocks (probably the first 4) are the ones that really matter, that the last 3 will have little impact on your score, since the computer would have found your ability level by then? Does this mean that the test tries to determine at which level you answer 60% correctly and 40% incorrectly, accounting for people guessing answers correctly?

Either way... how does that affect how you treat the exam? Its not like if its a CAT exam you will get a higher score by getting some questions wrong, so treating every single question seriously would be a good idea

 

[chaperone]

[...] so treating every single question seriously would be a good idea [...]

I guess it'd be more correct to say, "guessing every single question seriously would be a good idea" Just kidding!

 

[tfe]

Kaplan maintains that anywhere from 30 to 35 questions on the exam are included so that they can be pretested and evaluated for use in other exams. In addition, as many as 10 to 15 other questions may be eleminated from the scored pool after the exam results are reviewed.

Interesting, ifu - in case of adaptive tests the results are given to the examinee on the spot - it must be because of these 10-15 experimental Qs that you don't get the test score right away.

 

[Mehrdad1]

The USMLE is not an adaptive exam, and never has been. Dr. Daugherty debunks this myth at the Kaplan Live Lectures. He added, that it would actually be much more fair exam, if they decided to make it adaptive. However, they aren't concerned with making the exam more fair. In fact, many FMGs believe the USMLE is adaptive, because some of the nursing exams in the US are. Their friends talk to their friends, etc, etc. All leading to rumors that the USMLE is adaptive too, which it is not.

As the poster before said, we're relegated to being "guinea pigs for life."

 

[werd]

it's not adaptive. cheers.

 

[gift annuity]

I guess the verdict is in - not adaptive.

 

[courier]

I guess the verdict is in - not adaptive.

Are you being sarcastic - excuse my ignorance, but I really can't tell!

 

[omn]

No, I remember that the NBME specifically denied this a few years ago. I don't think this has changed.

Is there a way of finding out the truth of this ?

 

[justabanana]

.

 

[omn]

I'm trying to bumping it (and adding that I actually took the SAT when it was a paper test and I can't describe how scary a computer adaptive test sounded to a teenager in 1984 (that's apple 2 and commodore 64 days)... they really "sold" the idea 2 us... but I'm convinced USMLE is, CAT block to block not question to question)

Sent from my SM-N900V using SDN mobile