Is this actually confirmed?
Well the test is graded on a curve so yeah it's confirmed in theory. If there is a form that people are doing terrible on, I think you can expect cut off to be lower. 400 is about a 6 percentile meaning as long you do better than 6 percent of test takers you pass.
It is the way
Actually it's graded on a different type of curve than you're used to. It's graded on how you do with each question (and the difficulty of each question) rather than a cohort of test takers.
It is graded against the standard of a "minimally competent physician" - "would a minimally competent physician get this question correct?". Each question gets a certain value associated with it (i.e. not all questions are scored/counted the same). You need to accumulate a minimum amount of points to "pass". That minimum amount to pass raw score is then converted to the NBOME scores (ie 400 minimum to pass for COMLEX, 194 for USMLE)
So if your bank of questions contain a lot of easy questions, the number of questions needed to pass will be higher than if your bank has a lot of hard questions (easy/hard questions as determined by NBOME/NBME after it goes through psychometric validation). Getting 50% of the questions right may result in a passing score or failing score, depending on the types of questions (and level of difficulty) you get correct.
So it is theoretically possible for everyone taking the test to fail. It is theoretically possible for everyone taking the test to pass. However they are statistical anomalies and should either event occur (or the results do not fall within statistical prediction), then the NBOME (or NBME) will likely investigate to see if there were any errors in the process.
It's also why it takes so long to get results back - although your raw score is known immediately after you complete the test, it goes through a series of validation to make sure each question you answered was valid. If everyone gets an easy question wrong, it gets evaluate to see why and determine whether to include/exclude that question (and reset the scale if excluded).
Every few years, the passing criteria gets re-evaluated (often called re-centering) - which is why you suddenly see more people failing or passing compare to prior years after re-centering since the criteria for passing changed)
This process is used for NBOME, NBME (USMLE), and even the specialty board examinations (ABMS and their constituent boards)
Here's a Wikipedia link on the "science" behind psychometric testing
Psychometrics - Wikipedia
en.wikipedia.org
So are the high scores on level 3 because people score relatively lower in terms of raw score since many are far removed from med school material and in specialty residencies? Hence creating a generous curve (based on prior experimental questions).Actually it's graded on a different type of curve than you're used to. It's graded on how you do with each question (and the difficulty of each question) rather than a cohort of test takers.
It is graded against the standard of a "minimally competent physician" - "would a minimally competent physician get this question correct?". Each question gets a certain value associated with it (i.e. not all questions are scored/counted the same). You need to accumulate a minimum amount of points to "pass". That minimum amount to pass raw score is then converted to the NBOME scores (ie 400 minimum to pass for COMLEX, 194 for USMLE)
So if your bank of questions contain a lot of easy questions, the number of questions needed to pass will be higher than if your bank has a lot of hard questions (easy/hard questions as determined by NBOME/NBME after it goes through psychometric validation). Getting 50% of the questions right may result in a passing score or failing score, depending on the types of questions (and level of difficulty) you get correct.
So it is theoretically possible for everyone taking the test to fail. It is theoretically possible for everyone taking the test to pass. However they are statistical anomalies and should either event occur (or the results do not fall within statistical prediction), then the NBOME (or NBME) will likely investigate to see if there were any errors in the process.
It's also why it takes so long to get results back - although your raw score is known immediately after you complete the test, it goes through a series of validation to make sure each question you answered was valid. If everyone gets an easy question wrong, it gets evaluate to see why and determine whether to include/exclude that question (and reset the scale if excluded).
Every few years, the passing criteria gets re-evaluated (often called re-centering) - which is why you suddenly see more people failing or passing compare to prior years after re-centering since the criteria for passing changed)
This process is used for NBOME, NBME (USMLE), and even the specialty board examinations (ABMS and their constituent boards)
Here's a Wikipedia link on the "science" behind psychometric testing
Psychometrics - Wikipedia
