Step 1 P/F: Decision

[libertyyne]

---

Members don't see this ad.

Is determining who makes a great ENT or Neurosurgeon or IM fellowships from a memorization test that many take pre-clinicals valid? I don't know, but it's a question to be asked.

I hate what this does to low-tier MDs/DOs (I'm in a DO school), but the idea of making Step 1 P/F makes sense in the long-term.

You are acting like the same people who bust their behind on step one arent busting their behind on clinicals and step 2.

 

[AnatomyGrey12]

Is determining who makes a great ENT or Neurosurgeon or IM fellowships from a memorization test that many take pre-clinicals valid? I don't know, but it's a question to be asked.

I hate what this does to low-tier MDs/DOs (I'm in a DO school), but the idea of making Step 1 P/F makes sense in the long-term.

No, but none of the other available metrics do either....

 

[libertyyne]

This is the thing. The better decision would be to mandate that students have a Step 2 score before submission of ERAS. And if they want to de-emphasize Step 1, then average it with Step 2 CK. If you think Step 2 CK is worth more than Step 1, then do a weighted average.

But why in the world are we getting rid of step 1 altogether? Its absolutely absurd.

IM not sure if you read the part where if you score a 230 realistically your score falls between 218 and 242.

 

[Mass Effect]

I'm really hoping every sentiment said here is super exaggerated and everything ends up ok in the end 🙁

I was told that an MD will open all doors for me, and having my fate decided as a PCP before I even start bothers the hell out of me. Legit the reason I didn't even apply DO, I wanted to make sure I had all options available to me.

EVERYONE should go to med school knowing that they may end up a PCP. It doesn't matter if you're at Yale or Podunk Med School. PCP is always a possibility, no matter who you are.

 

[ace_inhibitor111]

Is determining who makes a great ENT or Neurosurgeon or IM fellowships from a memorization test that many take pre-clinicals valid? I don't know, but it's a question to be asked.

I hate what this does to low-tier MDs/DOs (I'm in a DO school), but the idea of making Step 1 P/F makes sense in the long-term.

They should make Step 2 CK pass/fail for the same reason, then. I think it’s a useless test for anyone NOT going into internal medicine. If I’m going into surgery, what use is it for me to know the STOPBANG score or the algorithm for treating CHF? Knowing the pathophysiology of CHF is at least slightly more helpful in terms of how to think about patients.

 

[Lawpy]

thoughts on making all steps pass/fail and have specialty specific exams instead?

 

[7331poas]

IM not sure if you read the part where if you score a 230 realistically your score falls between 218 and 242.

Then the response to that is to improve the confidence interval. Not to effectively get rid of the test.

 

[efle]

@efle Most people can’t abstract immediate visceral feelings from weighing in logically when it comes to a high performance examination. Step 1 P/F was likely going to be implemented because there were too many parties that perceived benefit from it without really rationalizing the outcomes. Top tier medical school students, medical schools in general, mediocre to poor performing students, and students who felt unprepared taking the examination were all invested in making this exam pass/fail because they all thought they would benefit from not having to worry about actually performing while reaping the benefits of being a “performer” in soft areas like LOR, sociability, and likability (Wizard of the Wards). There are medical students who actually believe that this will hurt students from top tier medical schools due to how it was spun to them with published articles from HYPSM schools coming to mind about how P/F benefits students from minority backgrounds. I think that SDN should be lauded for seeing through the low hanging fruit of P/F when most people go for immediate satisfaction over looking at the long term implications this will have on objective decision making.

I dunno, I have a strong score and removing it would only hurt my app. The specialty I'm most likely going after doesn't need a high score anyways, and since I'm not taking any research time, this isn't going to affect me at all. Yet I was still rooting for Pass/Fail because I read up on the history and design of the exam, and the stats of the exam (e.g. having a 16-point interval just to contain 60% confidence), and seeing what it was doing to the preclinical experience. Yes, this sucks for applicants who had poor college grades or bombed the MCAT and needed to use it as a second chance so they can match surgical subspecialties. But that vocal minority is not a valid reason to keep abusing and misusing a metric like this.

I mean hell, the NBME themselves had the greatest stake in keeping this exam scored. They make a BOATLOAD off selling their practice exam forms. Even they, after seeing all sorts of internal data that we are not privy to, had to do it.

Mark my words, they didn't want this either when they launched the INCUS ordeal last year. There's a lot behind the scenes (like the random sample survey results, or their internal data on test form performance/score creep) that made them feel this was necessary.

 

[topssure]

Changing scores retroactively beginning 2022 seems very illogical and I doubt it. Also they stated 2022 at the earliest. Changes as big as this usually takes longer than 2 years to implement. I think realistically 2024 will be the year for the change

 

[Mass Effect]

This is the thing. The better decision would be to mandate that students have a Step 2 score before submission of ERAS. And if they want to de-emphasize Step 1, then average it with Step 2 CK. If you think Step 2 CK is worth more than Step 1, then do a weighted average.

But why in the world are we getting rid of step 1 altogether? Its absolutely absurd.

They're not getting rid of it. You still have to pass it. But it really is the minimum you need to know and has (or used to have) a small amount of relevance to clinical medicine.

 

[Mkif586]

thoughts on making all steps pass/fail and have specialty specific exams instead?

Isn’t it how it works in some other countries?

 

[Mass Effect]

thoughts on making all steps pass/fail and have specialty specific exams instead?

Aren't those called shelf exams?

 

[Alkaidius]

EVERYONE should go to med school knowing that they may end up a PCP. It doesn't matter if you're at Yale or Podunk Med School. PCP is always a possibility, no matter who you are.

Careful, you are speaking in a forum where half the population would literally vomit if they learned they were relegated to a future in Primary. Not trying to diss anyone, just that this forum is an echo chamber of the most ambitious students in the small percentage of students that were able to even get this far.

 

[BoneWizard]

Will the minimum passing score change though?

 

[efle]

Then the response to that is to improve the confidence interval. Not to effectively get rid of the test.

There is no way to write a knowledge-check test across 2 years of content with a narrow interval. The only way tests like the SAT or MCAT retain tight intervals is by testing aptitude with novel content instead of primarily going after knowledge base. When they only get 1 or 2 questions to assess the strength of your knowledge base for entire categories, there's no way to avoid having a 30+ point interval for a good p value.

My personal p=.05 interval spanned from 230s to 270s. That is insane. This test was never designed to do head-to-head comparisons of applicants with a 240 vs 250, like what happens in reality.

 

[Mass Effect]

Careful, you are speaking in a forum where half the population would literally vomit if they learned they were relegated to a future in Primary. Not trying to diss anyone, just that this forum is an echo chamber of the most ambitious students in the small percentage of students that were able to even get this far.

I'm not going to hide reality just to feed into their fantasies.

 

[libertyyne]

Will the minimum passing score change though?

probably go down since there is less stress to be 250+

 

[gulli_97]

Eh I’m sure those wanting to go into surgical subspecialties, even those who are DO and low tier MDs, will still find a way. Whether that be through networking, research, or any other measure.

 

[Lawpy]

Aren't those called shelf exams?

they're still NBME though? i'm thinking like specialty boards but a preliminary version taken at school

 

[Mass Effect]

they're still NBME though? i'm thinking like specialty boards but a preliminary version taken at school

The ones I took were all NBME shelf exams.

 

[BoneWizard]

probably go down since there is less stress to be 250+

So what you're telling me is DO pass rates for Step 1 might actually go up if they have to study for COMLEX too?
That's amazing.

 

[gamerEMdoc]

Step 2 is more clinically useful. The problem is the timing of it all. The results aren't back until after you have already scheduled and done some of your AIs. It will be nearly impossible to pivot to something more/less competitive based on your scores. So if you want to do ortho, what if you are on your 3rd ortho AI and don't score above a 260? And how are fields that really stress board scores to determine who to let do an AI going to figure out who to let rotate? Sure that lets everyone get to apply for rotation, but are you doing yourself a favor rotating at a place that won't rank you once you get your step 2k back months later?

That's my biggest issue with this. Knowing your scores helps guide you to apply to fields you are competitive in, and to places you are competitive at. But by the time step 2 comes back, AIs are mostly over.

This will hurt students IMO. I'd rather have just seen them go all P/F for all exams and just de-emphasize scores all together.

 

[deleted831695]

You are acting like the same people who bust their behind on step one arent busting their behind on clinicals and step 2.

No, but none of the other available metrics do either....

They should make Step 2 CK pass/fail for the same reason, then. I think it’s a useless test for anyone NOT going into internal medicine. If I’m going into surgery, what use is it for me to know the STOPBANG score or the algorithm for treating CHF? Knowing the pathophysiology of CHF is at least slightly more helpful in terms of how to think about patients.

My apologies if that was poorly phrased. I'm just speculating the counterargument is that in theory it's better to gauge someone's competitiveness by their clinical grades and CK than a test (though some schools have shifted to taking it after third year) taken before clinicals. But you're right that it's imperfect, because clinical grades are subjective AF and the CK isn't the best written exam either. I don't like it for what it does to my fellow DOs gunning for something competitive.

 

[Lawpy]

The ones I took were all NBME shelf exams.

yeah but what about the residency boards but some of them taken in med school?

 

[efle]

If they have evidence that most medical students that take the USMLE wanted this change then that data needs to be publicly released. If this change was truly data driven and not with an agenda then it should be published by the end of the year. Let people analyze their methods, raw data, and conclusions.

Totally agree. Though, I can't imagine any agenda that would drive the NBME to do this if it was unpopular. They only gain from Step 1 Mania.

 

[Mass Effect]

yeah but what about the residency boards but some of them taken in med school?

Can't see this happening.

 

[Matthew9Thirtyfive]

Thank God I made it in before this travesty.

Welp, future M1s time to start maining step 2 decks from day 1 of M1.

My school takes step 1 in M3, so even though I’m class of 2023, I’ll probably end up taking the P/F version.

 

[efle]

Was this survey before or after they received their Step 1 score?

Before

 

[ace_inhibitor111]

There is no way to write a knowledge-check test across 2 years of content with a narrow interval. The only way tests like the SAT or MCAT retain tight intervals is by testing aptitude with novel content instead of primarily going after knowledge base. When they only get 1 or 2 questions to assess the strength of your knowledge base for entire categories, there's no way to avoid having a 30+ point interval for a good p value.

My personal p=.05 interval spanned from 230s to 270s. That is insane. This test was never designed to do head-to-head comparisons of applicants with a 240 vs 250, like what happens in reality.

I wonder how that interval is even estimated when nobody is allowed to take the test twice.

 

[AnatomyGrey12]

I mean hell, the NBME themselves had the greatest stake in keeping this exam scored. They make a BOATLOAD off selling their practice exam forms. Even they, after seeing all sorts of internal data that we are not privy to, had to do it.

Mark my words, they didn't want this either when they launched the INCUS ordeal last year. There's a lot behind the scenes (like the random sample survey results, or their internal data on test form performance/score creep) that made them feel this was necessary.

So just to be clear, your argument, without knowledge of this or having seen a shred of actual data, is that the NBME is being benevolent with this? That the same people who gave us Step 2 CS are only looking out for the medical students best interests...

If they have internal data driving this they are obligated to release it. If we don’t see any published data by the end of this year then there should be a very high suspicion of an agenda, even higher than currently exists.

 

[Mass Effect]

Before

Yeah, I don't give much credence to surveys given before score release.

 

[ace_inhibitor111]

Totally agree. Though, I can't imagine any agenda that would drive the NBME to do this if it was unpopular. They only gain from Step 1 Mania.

The AAMC is their biggest stakeholder. We’re still paying them to take a Pass/Fail Step 1 regardless.

 

[workaholic007]

anyone stating the previous score had to high of a variance - why didnt they just attempt to lower the varience of the score, make scoring tighter, or anything else rather than turning it pass/fail. logic doesnt add up

 

[Lawpy]

Can't see this happening.

why not

 

[efle]

I wonder how that interval is even estimated when nobody is allowed to take the test twice.

There's a good deal of statistical tricks out there. For example, randomly take half the items I answered and compare to the other random half of items. If I got exactly 135/140 on both halves, put me down as 100% agreement. Do this for a thousand test-takers and build a bell curve of the discrepancy.

 

[7331poas]

There is no way to write a knowledge-check test across 2 years of content with a narrow interval. The only way tests like the SAT or MCAT retain tight intervals is by testing aptitude with novel content instead of primarily going after knowledge base. When they only get 1 or 2 questions to assess the strength of your knowledge base for entire categories, there's no way to avoid having a 30+ point interval for a good p value.

My personal p=.05 interval spanned from 230s to 270s. That is insane. This test was never designed to do head-to-head comparisons of applicants with a 240 vs 250, like what happens in reality.

increase the length of the exam or make it multiple parts. Or change the style of the exam.

Do something that retains a student's ability to have upward mobility.

 

[Angus Avagadro]

I predict a number of schools return to a fully graded and ranked pre-clinical system.



Go take a gander at who the loudest and biggest proponents were of a P/F USMLE Step 1.... Almost universally the push came from individuals associated with large, highly ranked, and well known institutions.

Medical students supported the change also.
AMSA endorsed going to pass fail
I remember several threads on SDN endorsing the idea of pass fail Step.
Well, the game.has now changed. Students will now have to decide how to play the game well. It's the only game in town.
Students lobbying NBME should have presented a more comprehensive plan rather than just going to pass fail and letting NBME figure it out. There are always unintended consequences. I feel the DO students now have lost an important tool in distinguishing themselves. Rotation evals are very subjective. Now students will be stressed as they will be putting in long days on rotations AND studying for CK.Lastly, There is little time and less data on how to play the NEW Match Game. I. wish I had something positive to offer

 

[efle]

So just to be clear, your argument, without knowledge of this or having seen a shred of actual data, is that the NBME is being benevolent with this? That the same people who gave us Step 2 CS are only looking out for the medical students best interests...

If they have internal data driving this they are obligated to release it. If we don’t see any published data by the end of this year then there should be a very high suspicion of an agenda, even higher than currently exists.

My argument is that the NBME only stands to benefit, ergo if they are announcing it Pass/Fail (I was expecting only Quartiles), they sure as hell aren't doing it because of a 50-50 split of public comments. The way they behaved about Step 2 CS makes my point, they don't do something because it is right. They do what is profitable or what is necessary, and this certainly won't be a profitable change. Like I said though, completely agree about seeing data.

 

[ace_inhibitor111]

There's a good deal of statistical tricks out there. For example, randomly take half the items I answered and compare to the other random half of items. If I got exactly 135/140 on both halves, put me down as 100% agreement. Do this for a thousand test-takers and build a bell curve of the discrepancy.

Hold on, that doesn’t make sense. The proportion you got wrong is a fixed percentage. The probability of getting equal halves of a test correct assuming random assortment only depends on the number of questions.

 

[Mass Effect]

why not

Because specialty exams test on basic knowledge are generally much easier than the Steps. Asking them to do a med student version would result in the IM exam asking what CHF is.

 

[efle]

Yeah, I don't give much credence to surveys given before score release.

To the contrary, they wanted to know things like how it impacted your mental health. Not gonna get an unbiased answer after someone finds out they killed it or bombed it. They'd get nothing but "keep it scored!" from the high end and "make it Pass" from the low end, if they waited until after.

 

[efle]

Hold on, that doesn’t make sense. The proportion you got wrong is a fixed percentage. The probability of getting equal halves of a test correct assuming random assortment only depends on the number of questions.

Correct, the only way they could shrink their interval would be to make it into a multi-day test with half a thousand or more questions. And even then, the rate of interval shrink from adding items is not great.

 

[after p3]

@efle I honestly feel like the necessity was institutional. There is no way that medical schools could keep up to the quality of services like SketchyMedical or even have similar quality lecturers to professors like Goljan, Najeeb, Ryan, Sattar, or Williams. Students have been more recently vocal about noting how much they pay for medical school compared to how little they pay for any of these online lecture services. The fact that there are memes about wellness lectures and mandatory lectures scheduled right before big block exams compared to ones glorifying services like uWorld e.g. "What did people even do before uWorld?" indicated to me that schools were under increased scrutiny especially with the primary political campaign platforms being on student loans and tuition costs.

Some adcoms have been direct that the tuition model is really appropriated for schools to fund rotations in the third year and the amount of tuition students spend for M1 and M2 is really to fund their own clinical clerkships down the road. However, the idea that students are basically paying into schools that simply raise more barriers to access education rather than reducing them is what I saw as being a large pressure to turn Step 1 into P/F in order for schools to be "let off the hook" from the mounting pressure as they were being compared to free market services that were doing their job much better for a much cheaper price. I think that a lot of student input is negligible to the overall landscape. Polls are hardly exhaustive and are a poor metric in which to pivot, but when institutional pressure is to encourage a P/F system from prestigious to "lower tier" schools then there is no reason that the changes would no go through at some point in time.

 

[ace_inhibitor111]

Correct, the only way they could shrink their interval would be to make it into a multi-day test with half a thousand or more questions. And even then, the rate of interval shrink from adding items is not great.

But what I’m saying is they can’t use that method since it doesn’t really affect anything. The MCAT had 230 questions and apparently a tighter interval. The only thing I can think off is the NBME estimated the variance by comparing the answers of say a 260 scorer with that of a 220 scorer, and seeing if all higher scorers got that same question correct vs all the low scorers, etc. But this would overestimate the actual variance when testing a very large body of knowledge.

 

[CyrusHabs7695]

Since I don't feel like reading 69 pages of comments, will this effect the class of 2022 at all?

 

[libertyyne]

Since I don't feel like reading 69 pages of comments, will this effect the class of 2022 at all?

ya'll are doomed. jk its status quo unless residencies start testing changes before hand.

 

[Lawpy]

Because specialty exams test on basic knowledge are generally much easier than the Steps. Asking them to do a med student version would result in the IM exam asking what CHF is.

so make it a lot harder for stratifying?

 

[Mass Effect]

To the contrary, they wanted to know things like how it impacted your mental health. Not gonna get an unbiased answer after someone finds out they killed it or bombed it. They'd get nothing but "keep it scored!" from the high end and "make it Pass" from the low end, if they waited until after.

But everyone thinks they failed before score release.

 

[CyrusHabs7695]

ya'll are doomed. jk its status quo unless residencies start testing changes before hand.

I saw someones comment saying its possible we might get a 3 digit score when we take the test this summer, then when we apply to residency it could just be listed as a 'P', is that accurate?

 

[Mass Effect]

so make it a lot harder for stratifying?

And who monitors how hard it is, the NBME? LOL. The fact is that until you're in the specialty, you shouldn't be expected to know the nitpicky nuances of the field. At the same time, if you want to stratify students, it needs to be somewhat difficult. The specialty boards aren't the ones to strike that balance and the NBME is not going to forfeit the task to them.