Oh boy. Lots to get caught up on this thread for me. Sorry, some of these quotes are from Page 1
The impetus for change is the fact that what was originally intended to be a test of minimal foundational medical knowledge has evolved into an all-consuming monster, one that is causing significant harm to medical education and student health. In the past couple of years everyone has reached broad consensus that it's a serious problem, and the time has come to do something about it.
The fact that the exam is designed to test minimal knowledge for licensing, and that residency programs are now misusing it, is often stated. IMHO, this is ridiculous. Regardless of how it was initially developed, using it for something else is fine. Does a better step score = a better resident? More on that later. But there's no reason an exam can't fulfill two separate goals at the same time.
Whether "everyone has reached a broad consensus" is debatable, since no one has talked to program directors at all about this. IM programs that take mostly IMG's depend on Step scores, as the rest of the elements of their applications that you reference in other posts are mostly useless.
Only happy medium I can think of would be a broad-strokes categorization like High Pass (top 1/3rd), Pass (most of the remaining 2/3rds), and Fail. That would be a nice step down from the insane competitiveness of striving for the top ~15%, without making it worthless to work harder than the bare minimum in preclinical years. It would suck to narrowly miss the cutoff, but then again someone in the high 230s wasn't going to be very competitive for stuff like Derm and Ortho prior to the score change anyways.
This is being discussed -- either quartiles, or some other categorization. The "problem" as you mention is that you create an artificial cutoff -- a 234 is a "Pass" and a 235 is a "Pass with Honors", yet the two scores are really the same. One could argue that if PD's screen out everyone with a S1 score less than 240 then this really isn't any worse, but some people will win and some will lose with this type of system. I could live with it.
Capping would work I think only if there is a guarantee of matching. It's not really fair to make applicant A who has below-average stats only apply to 10 programs or whatever, since the more programs he applies to, the better his chance of matching. Not sure how you do that though, unless you did it like the military, where you list your top two choices of specialty and then your ranked list of locations, and someone takes you.
Capping applications is very controversial, it would mean that applicants would need to pick their applications very carefully. There have been suggestions floated that the number of apps be unlimited, but interviews be capped. And, if any limit in apps was placed there would be an exemption for couples -- I expect multiple medical students would "say" they are couple's matching just to get more apps, then not couples match in the NRMP.
Residency programs were capable of choosing highly qualified applicants in the decades before Step 1 became the be-all and end-all. The current arms race is largely a product of 1.) over-application to residency programs, and 2.) the further commercialization of the exam/exam prep industry.
I believe that they could make Step 1 P/F tomorrow and it would not have much effect on the outcome of the match. Programs would simply shift to other criteria that they already evaluate: Step 2 CK, clerkship grades, elective grades, shelf exams, sub-I grades, audition rotations, the MSPE, faculty LOR's, etc. Step 1 has never been the only piece of the puzzle, it's just the easiest one to obsess over.
I'd add 3) the growth of social media driving this insane behavior. Which is why I don't think that changing S1 to P/F will fix anything. medical students will pick the next thing -- probably S2 -- and just focus on that. Students will have "first aid for S2" on day one, and this insane focus on exam scores will stretch into the clinical years.
Fifteen years ago the average Step 1 score was 217, so an upper 230's was not slightly above average. Also, the exam did not report any specific score above a 245.
In the 1990's Part 1 was retired and USMLE Step 1 debuted, and thus began the slow but inexorable march toward our current state. The death knell occurred when program directors came under pressure to have 80% board passage rates in order to maintain ACGME accreditation, with Step 1 being seen as a good predictor of success on the speciality boards.
The key difference between the current situation and 10+ years ago isn't that Step 1 has really changed, or even that scores have increased. It's that residency programs no longer have the manpower to read all the applications they receive, so they use Step scores to whittle the number down to a manageable level. After that the rest of the package becomes more important.
This is somewhat anecdotal, but consistent across multiple program directors I know personally. The general feeling is that Step 1 scores have no real predictive power in terms of differentiating good residents from bad ones. One told me that the program's best residents are the ones with lower scores (meaning 220-240), because they tended to have strong interpersonal skills and could therefore function effectively in the clinical environment.
Scores about 245 were definitely reported. I took S1 >20 years ago, my score was above 245.
Another "fact" mentioned by student leaders is that step scores don't correlate with resident performance. This simply isn't true. Of course, nothing is a perfect predictor and some residents with high step scores have interpersonal problems, or lack team leadership, etc. But in general, residents with higher scores in my program do better than those with lower scores.
The third "fact" I'm often told is that as long as someone has a step score >211 they are guaranteed to pass the boards. This is ridiculously false.
The research game has also blown up into a monster of its own. For the most competitive specialties the AVERAGE numbers of posters/pubs on ERAS is 15-18. My friends aiming for things like academic neurosurg spend all their time finding ways to fluff up their research list, like doing a million mindless chart review data extractions for middle authorship, or presenting the same stupid small summer project at 8 different conferences. Having just a couple good longitudinal involvements in projects is perceived as killing your chances. Research years are also becoming a lot more common.
These numbers are up SEVERAL FOLD from just ten years ago. It's all madness and clearly a far cry from actual value added to their candidacy
This is driven IMHO by statistics, much like application inflation. If I was told that the average number of apps last year was 12 and I had to apply, you know how many apps I'd submit? Probably 14-15. I like being above average. Lake Wobegon and all that. So then next year the average is 14. Surprise! Same thing happens with everything else -- everyone is striving to be above average, and averages increase. It's inflation.
We're really hampered by the fact that the NBME has never divulged its system for generating a three digit score. Step 1 is an odd test for a number of reasons. One of them is that its ostensible purpose is to evaluate whether a taker has a minimum fund of medical knowledge. It would therefore make sense for Step 1 to be a criterion-referenced test. But it's not. It's a norms-referenced test, build around the notion that 5-6% of US allopathic students should fail it on the first attempt.
I'm mixed on this. I agree the USMLE hasn't published it's scoring rubric. I also agree that there should be a criterion cutoff for passing. But just because 5% fail each year, one would expect that the same percentage of people each year would not meet that criterion. Or, perhaps, the criterion increases over time as there is more to know.
I suspect a significant driver, which happens to be more recent, is the proliferation of self-assessment exams and the inception of Reddit. The combination of these two seems to work even the most stoic student into a lather.
Totally agree. Which raises the question of whether changing S1 to P/F really fixes the problem, or only treats the symptom.
If Step 1 goes P/F, the optional nature of Step 2 would vanish overnight. It has already become the fourth most commonly cited factor for granting an interview, with an average rating of 4.0 (vs. 4.1 for Step 1).
Totally agree. So the exam insanity would just shift to S2. And, now there's only one exam. If you screw up S1, you can always take S2 early and try to do better. If S1 is P/F, you get one chance at S2, and by the time you have a score you're probably already applying.
Would making Step 1 P/F nearly destroy US citizens' chances at matching from an IMG?
No. The number of spots, and the number of applicants remains the same. IMg's will still get spots, there are more spots than US grads.
probably but the path to residency for IMGs has been narrowing every single year simply because the number of medical school graduates is increasing faster than the number of residency slots.
This isn't true, at least not yet. We might be at an inflection point now, though.
-----
Overall, I understand why student leaders are frustrated with the current state of S1. Med students see S1 as their primary focus, and anything that isn't directly related to S1 is blown off. Anything non Step targeted that medical schools try to plug into their curriculum fails. And there's this sense (somewhat correct) that the S1 result defines their future.
Making S1 quartiled (or similar) might help, although would students really study less, how you you be certain you were in the top 25%
Making S1 P/F will just put that pressure on S2. All students will be required to pass S2 pior to applications, and that score will become paramount.
Make both S1 and S2 P/F, and programs will add specialty specific exams. Which will be an added cost, and more studying and stress.
Medical schools could report true quartiles in ERAS -- that might take the pressure off S1. But you're not allowed to put 65% of your students in the first "quartile", which plenty of schools do. Just telling us all of your students have met their milestones / EPA's / whatever and that we need to do a "holistic" review is impractical.
Dealing with app inflation is a whole topic on it's own.
... and I'm off to a forum with the NBME. Should be a party!
