An interesting discussion.
In general, the fail rate on S1 has been around 5-6% of first time MD students. We have clearly seen a slow increase in S1 means, without a big change in the fail rate -- due to slow increases in the minimum passing score.
The USMLE used to claim that scores were equivalent across years -- that a 200 today was equivalent to a 200 fifteen years ago. They no longer claim this, now state that scores more than 3-4 years apart shouldn't be compared. The USMLE is not transparent about what the scaled scores represent. Has the relationship between a score and percentage questions correct remained the same? This would be the simplest explanation, but isn't commented upon in their documentation. They do state that to currently pass you need to get approx 60% of the questions correct - and this is independent of the step, so I think it's unlikely that the scaled score <=> percent correct has remained stable over time.
The thought that equivalent scores over time represent equivalent performance is hard to believe. The exam has changed dramatically over time -- in content and in style -- such that it's impossible to compare how someone who passed the exam years ago would do with today's exam. Are today's students really that much smarter than those 15 years ago that they all would pass now? Also seems suspect.
Theoretically the minimum pass is set by some set of experts. A common method is the Angoff process, but there are others. As the content has changed, and as time has moved on, it's also possible that experts expect more from students and hence the true minimum pass level may have increased -- whether that's good (holding physicians to a higher standard) or not (needlessly preventing some from being physicians) is a matter of debate.
Ultimately before we panic about this, let's see what the fail rate is. I've seen plenty of predictions on SDN that have never come to fruition. Right now this is all rumor and hearsay. If the fail rate has increased, then it raises interesting questions as to whether students should just study harder and do better, or whether the pass level is too high. Politically I agree that lowering the pass rate seems untenable -- especially as then those with previously-failing-but-now-passing scores would be very bitter.
No matter what is done, the pass line is arbitrary. The person who scores a barely passing score and the one who gets one more question incorrect and fails -- their performances are not significantly different. That's the nature of any cutoff.
The comment above about COMLEX and USMLE is very interesting, I've pointed it out below. If you look at any study comparing USMLE to COMLEX performance, the regression line always shows that a minimally passing COMLEX score is a failing USMLE score. Which is "correct" is also a matter of debate.
