Are we about to witness a return to high Step 1 fail rates?

[efle]

We have a much more comprehensive knowledge of physiology than attendings who were in school at that time. We have way more meds and way more treatments. In the 90s we didn’t give beta blockers in heart failure and you essentially always admitted uncomplicated pneumonia. We’re expected to hit wards knowing a ton of physiology and pharm that was poorly understood then or (in the case of pharm) didn’t even exist. So yeah, I think the knowledge bar is higher now for what we’re expected to know when we hit wards. The standard is higher.

We definitely learn clinically irrelevant minutiae, but as I’ve already said, no one’s failing based on that if they know the important stuff. As an aside, the micro minutiae was nothing on step 1 compared to comlex.

Now something we’ll both likely agree on is that when someone hit wards in 3rd year 20 years ago with knowledge deficits, the information was drilled into them. That’s not the case anymore. So I firmly believe our attendings knew the phys and pharm they we’re supposed to know by the end of med school. But they learned it during rotations instead of showing up with it all memorized as is the norm now.

This would be plausible to me, if it wasn't the case that knowledge used daily moved away from Step 1 material over time. Medicine shelf has a lot of Step 1 in it. Step 2 CK had a decent amount of overlap. Step 3 felt like a very different set of knowledge, way more geared towards management decisions. I'm at the tail end of an inpatient-only IM year now, and looking back at Sketchy slides, I honestly couldn't tell you even half of that info anymore, but I feel far more comfortable making decisions about my patients' care. I do believe the wards used to beat important info into clerks much more than modern times. But I really don't think that info was the kind of stuff we cram and re-memorize during Step 1 dedicated to move our scores up 30 points. It was all about management decisions, telling you what you messed up in your planned orders, or missed on history and exam, things that would make you a better intern - things that attendings actually know a lot better than their med students.

---

Members don't see this ad.

 

[Ho0v-man]

I thought this way in med school and residency, especially in talking with my FP uncle.

But then I got a copy of my grandfather's Grey's Anatomy from the 40s and talks to said uncle and my OB/GYN father-in-law. That anatomy book had significantly more detail than my anatomy textbook from 2005. And the physical exam skills they learned in the 70s were significantly more involved that what I learned. The amount of stuff that had to be learned I don't believe was that different. but the focus is on whatever is known at the time. Fewer medications were around in the 70s, so more emphasis on anatomy and exam skills.

I almost brought the anatomy thing up but was worried my experiences were just too anecdotal. Yes. Before the USMLE gods decided that anatomy was low-yield for whatever reason, anatomy was much much worse from my discussions with attendings who trained even in the 90s. The FM trained ER docs I worked with before med school could go into a ton of depth like it was nothing. I was only on that level right before a lab practical.

 

[efle]

I thought this way in med school and residency, especially in talking with my FP uncle.

But then I got a copy of my grandfather's Grey's Anatomy from the 40s and talks to said uncle and my OB/GYN father-in-law. That anatomy book had significantly more detail than my anatomy textbook from 2005. And the physical exam skills they learned in the 70s were significantly more involved that what I learned. The amount of stuff that had to be learned I don't believe was that different. but the focus is on whatever is known at the time. Fewer medications were around in the 70s, so more emphasis on anatomy and exam skills.

Physical exams are easy these days, you just hit copy-forward and then don't actually do one (jk...kinda)

 

[Lem0nz]

I thought this way in med school and residency, especially in talking with my FP uncle.

But then I got a copy of my grandfather's Grey's Anatomy from the 40s and talked to said uncle and my OB/GYN father-in-law. That anatomy book had significantly more detail than my anatomy textbook from 2005. And the physical exam skills they learned in the 70s were significantly more involved that what I learned. The amount of stuff that had to be learned I don't believe was that different. but the focus is on whatever is known at the time. Fewer medications were around in the 70s, so more emphasis on anatomy and exam skills.

I would concur with this. From a surgeon’s perspective, the level of detail of anatomy has fallen in recent years precipitously. I am frightened at how much anatomy I didn’t know in fellowship, and how much MORE I didn’t know as an attending and had to very quickly catch up on.

Our treatments have become more robust through smaller holes and our ability to recognize and recover patients has made it possible to do more with far less knowledge. It is why the learning curve is not nearly what it once was. We are not smarter nor faster, we just got more efficient with the motions we take.

My own co-chiefs are straight up frightened when I send them pictures or descriptions of what I’m doing and I don’t blame them. 3-4 years ago I would not have been able to do this. I can absolutely see the value of the level of detail from surgeons trained 20 years ago.

It’s all very strange. My two cents is that it has led to a frightening loss of innovation and creativity at a time when we need it most in medicine, and particularly in surgery as people live longer with more complex problems. But it does make a lot of cool opportunity for those who don’t want to take the well traveled road.

 

[Ho0v-man]

This would be plausible to me, if it wasn't the case that knowledge used daily moved away from Step 1 material over time. Medicine shelf has a lot of Step 1 in it. Step 2 CK had a decent amount of overlap. Step 3 felt like a very different set of knowledge, way more geared towards management decisions. I'm at the tail end of an inpatient-only IM year now, and looking back at Sketchy slides, I honestly couldn't tell you even half of that info anymore, but I feel far more comfortable making decisions about my patients' care. I do believe the wards used to beat important info into clerks much more than modern times. But I really don't think that info was the kind of stuff we cram and re-memorize during Step 1 dedicated to move our scores up 30 points. It was all about management decisions, telling you what you messed up in your planned orders, or missed on history and exam, things that would make you a better intern - a.k.a. things that attendings actually know a lot better than their med students.

I guess we’re just going to disagree like we usually do. I remember those old NBMEs they don’t offer anymore had tons of minutiae compared to my actual Step 1. Those exams were NOT more clinically relevant than the exam I took.

I don’t think the point of these exams is to make sure you have common treatment algorithms memorized. It’s to make sure you have the basic science background to know why the algorithm is the algorithm and that when you’re in an unfamiliar situation to be able to assess the data points you have and work toward a reasonable plan. Learning how to apply this info is the goal of third and fourth year.

I’ve worked with midlevels with years of experience during my intern year who just don’t have this basic science background to fall back on when they’re stuck. Even if you don’t remember it crystal clear, you know the information exists and how to find it and apply it to you’re situation.

Yes, I won’t lose any sleep if we stop memorizing gene names or the shapes of a viral capsid either. And being able to ignore that stuff and still comfortably pass was actually a great argument for pass/fail. But that isn’t the majority of the test.

 

[Ho0v-man]

I'm just curious, to the doctors on here, do you think you could pass step 1 if you took it tomorrow? Kinda curious what you guys would get in an NBME or free 120. Someone take me up on that challenge, there will be no prizes.

FWIW, I took the free 120 last night and got an 80%. Granted, I did it untimed on my phone and a lot of it during commercial breaks. So not as stressed as y’all when you take it. Kinda had to shake off some cobwebs during the first block lol. I took step 1 three years ago.

 

[VA Hopeful Dr]

FWIW, I took the free 120 last night and got an 80%. Granted, I did it untimed on my phone and a lot of it during commercial breaks. So not as stressed as y’all when you take it. Kinda had to shake off some cobwebs during the first block lol. I took step 1 three years ago.

Wait, what's this now?

 

[introp445]

I feel for people in the current step 1 scenario. As someone who took it in the final year of normal scoring, my classmates and I still had the threat of the score and obviously studied all of preclinical to do all we could on the test. As it’s been said before, that is why for so many of us the idea of failing is essentially a nonissue. If you prepare that way there is such a small chance you’ll fail. Now students “just have to pass” which to older students and those that have been through it may seem easy, however as someone that did decently well on step 1 and 2, I can say step 1 was probably the most difficult test of my career to date. Many will agree with me. The NBME, in trying to alleviate stress by doing this, really ramped it up in my opinion. I wouldn’t be surprised if it is scored again in the future.

 

[EvoMed]

This year was a a year of growing pains. Without the crushing existential pressure of your fate relying on a single test, students didn't start studying 1 year in advance as students in previous years have. However, many students underestimated the challenge of just getting to Pass, thinking that 1 month of preparation (after competently getting through MS1 and MS2) would be sufficient to hit the pass mark. However, the sheer breadth, level of detail, and of course some idiosyncrasies of the exam make it such that at least 3-4 months of preparation is really neccesary.

I think going P/F was a good call, and I also agree that lowering the pass threshold is counterproductive. Nevertheless, this years class was the one to take the smack in the face of the wake-up call that P/F does not mean wait until 4 weeks before to prepare. I think this message will be passed to future years, so next years class will be the ones to benefit from the transition to P/F

 

[MDMechEngr]

I have not taken Step 1 yet. I thought the Surgery Shelf Exam was the hardest test I ever took in my entire life up to now. Thought I bombed it. My score came back at 99th percentile. Step 1 failure rate will remain relatively the same, unless the other 40% of DO students decide to take Step 1.

 

[tpjunkie]

My two cents - a good chunk of this is due to COVID. Granted, I graduated Med school 9 years ago, but hear me out. I did an SMP prior to medical school, which I enjoyed and obviously, did the trick for me. My wife just graduated from the same program. Covid started right after her first semester, and there was a noticeable drop off in teaching quality and resources available for her, compared to what I had access to. Unsurprisingly, similar effects were seen at the Med school it is a part of. Things sound like kind of a mess there administratively (worse than baseline for the place, which was already a little iffy), which is obviously going to have an effect on academics. This is more direct info based on my conversations with current Med students there (ms1 and 2s) that I play hockey with. They seem to have been given even more dedicated pre-mcat study time than in my day as well.

 

[Ho0v-man]

Wait, what's this now?

Free practice exam for step 1. I don’t know what this means because I think it’s different that the one I took, but iirc that was a correlated with a comfortably passing score when I took a previous version of it.

 

[32valveMD]

Step 1 had a 98% first time pass rate in 2021. If someone wants to find and post the granular data that shows what it would have been with a 2 point increase i’m all ears. Otherwise, it’s not the test, it’s the attitude towards studying for it now that it’s not scored but rather pass fail. Docs that took the exam in the 90’s are now at least In their mid 50’s and are decreasing in workforce numbers. The bulk of attending physicians practicing right now are younger than that and are products of competitive scored step exams. There is a worth to that knowledge base and it’s already been discounted by making step 1 pass / fail. Let the experiment run before we go and change the goalposts again.

 

[DrZoidbergDO]

Tested today, got one question right because I knew the feces of a specific bug species causes a certain pathology (thanks sketchy) and got another because I knew the description of the the histopathology (description, wasn't even shown the slide) of a parasitic infection (thanks uworld) - if I ever use this information again, I'll eat my sock.

Lol I took it the same day as you and I know exactly which questions you're talking about haha. Honestly those were gimmes.

As for the increased risk of failing, I'm not surprised. I personally started board prepping hard in December and I was still not scoring amazingly high on practice tests. I know a lot of people at my school who didn't take it seriously and are now desperately trying to pass comlex, let alone step. Also I've had people in the class under me say why bother with studying that hard for just PF, so I think this will last a few years at least before a new equilibrium is reached.

 

[efle]

Free practice exam for step 1. I don’t know what this means because I think it’s different that the one I took, but iirc that was a correlated with a comfortably passing score when I took a previous version of it.

Yea iirc a 70% is passing equivalent so just a few more years you'll be in the danger zone. I'd guess I'm already there

 

[Lem0nz]

Let's do SDN's first ever IRB approved (mods can be the IRB) prospective study on medical education. Do a bunch of hobos taking a 120 practice test of Step 1 correlate to the state of affairs of medical education in 2022? Hypothesis - it does, medical students are a bunch of sissys. Null hypothesis - it doesn't, old doctors are really dumb and need to stay in their lane.

Sufficient power: NAPD does it and we use his results to be reflective of the entire study and ignore all other data points.

Exclusion criteria: Lawpy is excluded because he has too many opinions on everything.

 

[Ho0v-man]

Yea iirc a 70% is passing equivalent so just a few more years you'll be in the danger zone. I'd guess I'm already there

You mean when we’re both *** deep in radiology residency? Yeah definitely lol.

But I took it mostly to make sure I wasn’t misremembering that stuff. Yeah there were some obscure cell receptor things I can’t remember. Again, stuff like that is a great reason to advocate for p/f.

But most of it is honestly the training wheels version of what you need to be a doctor. And it’s information you’re expected to know on the spot in 3rd year

“Which tumor secretes this hormone?”
“Which type of study is this?”
“Risk factors for this condition?”

People can’t pass this after 2 years of preclinical and just going through uworld? Gimme a break!


BUT the experiment isn’t over yet. We’ll have to see how this translates into clinical performance. After all, now that these students haven’t had to worry about step through all of preclinical, I’m sure they spent that extra time learning important clinical skills and will all be wizards of the wards. Surely they didn’t all just study less and not do anything meaningful in their downtime /s

 

[Goro]

FWIW, I took the free 120 last night and got an 80%. Granted, I did it untimed on my phone and a lot of it during commercial breaks. So not as stressed as y’all when you take it. Kinda had to shake off some cobwebs during the first block lol. I took step 1 three years ago.

Can you share that link???

 

[2020NonTrad]

Can you share that link???

NBME Orientation

orientation.nbme.org

Choose Step 1 All Blocks for the "Free 120" questions

 

[idkmed]

Just adding some more (anecdotal) evidence, but I was talking to the head of academic counseling at my school and she said she's part of a group of administrators in similar roles in various role across the country and almost universally schools are reporting significant increases in students not passing or requesting a delay of their dedicated period due to not passing their practice exams.

 

[efle]

You mean when we’re both *** deep in radiology residency? Yeah definitely lol.

But I took it mostly to make sure I wasn’t misremembering that stuff. Yeah there were some obscure cell receptor things I can’t remember. Again, stuff like that is a great reason to advocate for p/f.

But most of it is honestly the training wheels version of what you need to be a doctor. And it’s information you’re expected to know on the spot in 3rd year

“Which tumor secretes this hormone?”
“Which type of study is this?”
“Risk factors for this condition?”

People can’t pass this after 2 years of preclinical and just going through uworld? Gimme a break!


BUT the experiment isn’t over yet. We’ll have to see how this translates into clinical performance. After all, now that these students haven’t had to worry about step through all of preclinical, I’m sure they spent that extra time learning important clinical skills and will all be wizards of the wards. Surely they didn’t all just study less and not do anything meaningful in their downtime /s

Haha remember when someone at the NBME suggested pass fail would mean med students just watch more Netflix? Haha he sure looked foolish (no /s)

I just do not believe there is any difference in people's smarts or willingness to learn medicine. They just aren't learning UFAPS for 18 months and calling that "medicine" like our generation did.

 

[sloop]

What I’m saying is stop asking us to diagnose mitochondrial disease or Cru de chat syndrome or differentiate between papillary and medullary thyroid disease lol. How does that help anybody?. Just test us on high yield stuff like heart failure or causes of respiratory failure, this is supposed to be a basic competency test. The Step 1 used to resemble the old NBMEs with a short paragraphs instead of long intricate IQ test type questions also

So, a race to the bottom?

 

[arjusmc]

One thing that is not discussed is what students would do to get high scores and in turn drive the score creep. UFAP is one thing however many students would completely disregard class and sit and do Anki all day. Memorizing fact after fact without context. The gold standard is the anking deck, which in the latest edition has 42k flash cards. At least if you’re watching boards and beyond, pathoma and reading FA you’re getting context. I believe the idea of the third order question is to determine if the student can use basic knowledge to extrapolate what would happen in a situation, instead anking would have you memorize every situation. P/F med schools made this an attractive option in years past.

Im not saying that all that used anking don’t understand the material but a high step score doesn’t prove that they do. I don’t mean to say Anki is bad, I use it, I made my own cards or unlocked ones relevant to lectures. It’s just how it was used that is problematic.

There was probably an underestimation on multiple parts, to include how long you’d have to prep to pass step 1 and how much students scores would drop overall, due to not sitting in front of a computer hitting a space bar for 14 hours a day for 18 months straight.

 

[arjusmc]

We have a much more comprehensive knowledge of physiology than attendings who were in school at that time. We have way more meds and way more treatments. In the 90s we didn’t give beta blockers in heart failure and you essentially always admitted uncomplicated pneumonia. We’re expected to hit wards knowing a ton of physiology and pharm that was poorly understood then or (in the case of pharm) didn’t even exist. So yeah, I think the knowledge bar is higher now for what we’re expected to know when we hit wards. The standard is higher.

We definitely learn clinically irrelevant minutiae, but as I’ve already said, no one’s failing based on that if they know the important stuff. As an aside, the micro minutiae was nothing on step 1 compared to comlex.

Now something we’ll both likely agree on is that when someone hit wards in 3rd year 20 years ago with knowledge deficits, the information was drilled into them. That’s not the case anymore. So I firmly believe our attendings knew the phys and pharm they we’re supposed to know by the end of med school. But they learned it during rotations instead of showing up with it all memorized as is the norm now.

Took comlex level 1 yesterday and came ready with all my micro bioterrorism knowledge to save the world and wasn’t disappointed.

 

[DOVinciRobot]

Took comlex level 1 yesterday and came ready with all my micro bioterrorism knowledge to save the world and wasn’t disappointed.

Ah yes, the Gimpel special

 

[Ho0v-man]

Took comlex level 1 yesterday and came ready with all my micro bioterrorism knowledge to save the world and wasn’t disappointed.

I even had one on level 3. Both times I just didn’t bother and just clicked randomly and moved on. I just can’t with comlex. I always lose my patience.

 

[arjusmc]

I even had one on level 3. Both times I just didn’t bother and just clicked randomly and moved on. I just can’t with comlex. I always lose my patience.

I had one at around question 340 or so, I may have had a similar response.