Step 1 P/F: Decision

This forum made possible through the generous support of SDN members, donors, and sponsors. Thank you.
Members don't see this ad.
Natural Killer Cell said:
Is determining who makes a great ENT or Neurosurgeon or IM fellowships from a memorization test that many take pre-clinicals valid? I don't know, but it's a question to be asked.

I hate what this does to low-tier MDs/DOs (I'm in a DO school), but the idea of making Step 1 P/F makes sense in the long-term.
Click to expand...
You are acting like the same people who bust their behind on step one arent busting their behind on clinicals and step 2.
 
Natural Killer Cell said:
Is determining who makes a great ENT or Neurosurgeon or IM fellowships from a memorization test that many take pre-clinicals valid? I don't know, but it's a question to be asked.

I hate what this does to low-tier MDs/DOs (I'm in a DO school), but the idea of making Step 1 P/F makes sense in the long-term.
Click to expand...

No, but none of the other available metrics do either....
 
7331poas said:
This is the thing. The better decision would be to mandate that students have a Step 2 score before submission of ERAS. And if they want to de-emphasize Step 1, then average it with Step 2 CK. If you think Step 2 CK is worth more than Step 1, then do a weighted average.

But why in the world are we getting rid of step 1 altogether? Its absolutely absurd.
Click to expand...
IM not sure if you read the part where if you score a 230 realistically your score falls between 218 and 242.
 
Jappa said:
I'm really hoping every sentiment said here is super exaggerated and everything ends up ok in the end 🙁

I was told that an MD will open all doors for me, and having my fate decided as a PCP before I even start bothers the hell out of me. Legit the reason I didn't even apply DO, I wanted to make sure I had all options available to me.
Click to expand...

EVERYONE should go to med school knowing that they may end up a PCP. It doesn't matter if you're at Yale or Podunk Med School. PCP is always a possibility, no matter who you are.
 
Natural Killer Cell said:
Is determining who makes a great ENT or Neurosurgeon or IM fellowships from a memorization test that many take pre-clinicals valid? I don't know, but it's a question to be asked.

I hate what this does to low-tier MDs/DOs (I'm in a DO school), but the idea of making Step 1 P/F makes sense in the long-term.
Click to expand...

They should make Step 2 CK pass/fail for the same reason, then. I think it’s a useless test for anyone NOT going into internal medicine. If I’m going into surgery, what use is it for me to know the STOPBANG score or the algorithm for treating CHF? Knowing the pathophysiology of CHF is at least slightly more helpful in terms of how to think about patients.
 
libertyyne said:
IM not sure if you read the part where if you score a 230 realistically your score falls between 218 and 242.
Click to expand...

Then the response to that is to improve the confidence interval. Not to effectively get rid of the test.
 
after p3 said:
@efle Most people can’t abstract immediate visceral feelings from weighing in logically when it comes to a high performance examination. Step 1 P/F was likely going to be implemented because there were too many parties that perceived benefit from it without really rationalizing the outcomes. Top tier medical school students, medical schools in general, mediocre to poor performing students, and students who felt unprepared taking the examination were all invested in making this exam pass/fail because they all thought they would benefit from not having to worry about actually performing while reaping the benefits of being a “performer” in soft areas like LOR, sociability, and likability (Wizard of the Wards). There are medical students who actually believe that this will hurt students from top tier medical schools due to how it was spun to them with published articles from HYPSM schools coming to mind about how P/F benefits students from minority backgrounds. I think that SDN should be lauded for seeing through the low hanging fruit of P/F when most people go for immediate satisfaction over looking at the long term implications this will have on objective decision making.
Click to expand...
I dunno, I have a strong score and removing it would only hurt my app. The specialty I'm most likely going after doesn't need a high score anyways, and since I'm not taking any research time, this isn't going to affect me at all. Yet I was still rooting for Pass/Fail because I read up on the history and design of the exam, and the stats of the exam (e.g. having a 16-point interval just to contain 60% confidence), and seeing what it was doing to the preclinical experience. Yes, this sucks for applicants who had poor college grades or bombed the MCAT and needed to use it as a second chance so they can match surgical subspecialties. But that vocal minority is not a valid reason to keep abusing and misusing a metric like this.

I mean hell, the NBME themselves had the greatest stake in keeping this exam scored. They make a BOATLOAD off selling their practice exam forms. Even they, after seeing all sorts of internal data that we are not privy to, had to do it.

Mark my words, they didn't want this either when they launched the INCUS ordeal last year. There's a lot behind the scenes (like the random sample survey results, or their internal data on test form performance/score creep) that made them feel this was necessary.
 
Changing scores retroactively beginning 2022 seems very illogical and I doubt it. Also they stated 2022 at the earliest. Changes as big as this usually takes longer than 2 years to implement. I think realistically 2024 will be the year for the change
 
7331poas said:
This is the thing. The better decision would be to mandate that students have a Step 2 score before submission of ERAS. And if they want to de-emphasize Step 1, then average it with Step 2 CK. If you think Step 2 CK is worth more than Step 1, then do a weighted average.

But why in the world are we getting rid of step 1 altogether? Its absolutely absurd.
Click to expand...

They're not getting rid of it. You still have to pass it. But it really is the minimum you need to know and has (or used to have) a small amount of relevance to clinical medicine.
 
Mass Effect said:
EVERYONE should go to med school knowing that they may end up a PCP. It doesn't matter if you're at Yale or Podunk Med School. PCP is always a possibility, no matter who you are.
Click to expand...

Careful, you are speaking in a forum where half the population would literally vomit if they learned they were relegated to a future in Primary. Not trying to diss anyone, just that this forum is an echo chamber of the most ambitious students in the small percentage of students that were able to even get this far.
 
7331poas said:
Then the response to that is to improve the confidence interval. Not to effectively get rid of the test.
Click to expand...
There is no way to write a knowledge-check test across 2 years of content with a narrow interval. The only way tests like the SAT or MCAT retain tight intervals is by testing aptitude with novel content instead of primarily going after knowledge base. When they only get 1 or 2 questions to assess the strength of your knowledge base for entire categories, there's no way to avoid having a 30+ point interval for a good p value.

My personal p=.05 interval spanned from 230s to 270s. That is insane. This test was never designed to do head-to-head comparisons of applicants with a 240 vs 250, like what happens in reality.
 
alkaidius said:
Careful, you are speaking in a forum where half the population would literally vomit if they learned they were relegated to a future in Primary. Not trying to diss anyone, just that this forum is an echo chamber of the most ambitious students in the small percentage of students that were able to even get this far.
Click to expand...

I'm not going to hide reality just to feed into their fantasies.
 
Step 2 is more clinically useful. The problem is the timing of it all. The results aren't back until after you have already scheduled and done some of your AIs. It will be nearly impossible to pivot to something more/less competitive based on your scores. So if you want to do ortho, what if you are on your 3rd ortho AI and don't score above a 260? And how are fields that really stress board scores to determine who to let do an AI going to figure out who to let rotate? Sure that lets everyone get to apply for rotation, but are you doing yourself a favor rotating at a place that won't rank you once you get your step 2k back months later?

That's my biggest issue with this. Knowing your scores helps guide you to apply to fields you are competitive in, and to places you are competitive at. But by the time step 2 comes back, AIs are mostly over.

This will hurt students IMO. I'd rather have just seen them go all P/F for all exams and just de-emphasize scores all together.
 
libertyyne said:
You are acting like the same people who bust their behind on step one arent busting their behind on clinicals and step 2.
Click to expand...

AnatomyGrey12 said:
No, but none of the other available metrics do either....
Click to expand...

ace_inhibitor111 said:
They should make Step 2 CK pass/fail for the same reason, then. I think it’s a useless test for anyone NOT going into internal medicine. If I’m going into surgery, what use is it for me to know the STOPBANG score or the algorithm for treating CHF? Knowing the pathophysiology of CHF is at least slightly more helpful in terms of how to think about patients.
Click to expand...

My apologies if that was poorly phrased. I'm just speculating the counterargument is that in theory it's better to gauge someone's competitiveness by their clinical grades and CK than a test (though some schools have shifted to taking it after third year) taken before clinicals. But you're right that it's imperfect, because clinical grades are subjective AF and the CK isn't the best written exam either. I don't like it for what it does to my fellow DOs gunning for something competitive.
 
AnatomyGrey12 said:
If they have evidence that most medical students that take the USMLE wanted this change then that data needs to be publicly released. If this change was truly data driven and not with an agenda then it should be published by the end of the year. Let people analyze their methods, raw data, and conclusions.
Click to expand...
Totally agree. Though, I can't imagine any agenda that would drive the NBME to do this if it was unpopular. They only gain from Step 1 Mania.
 
slowthai said:
Thank God I made it in before this travesty.

Welp, future M1s time to start maining step 2 decks from day 1 of M1.
Click to expand...

My school takes step 1 in M3, so even though I’m class of 2023, I’ll probably end up taking the P/F version.
 
efle said:
There is no way to write a knowledge-check test across 2 years of content with a narrow interval. The only way tests like the SAT or MCAT retain tight intervals is by testing aptitude with novel content instead of primarily going after knowledge base. When they only get 1 or 2 questions to assess the strength of your knowledge base for entire categories, there's no way to avoid having a 30+ point interval for a good p value.

My personal p=.05 interval spanned from 230s to 270s. That is insane. This test was never designed to do head-to-head comparisons of applicants with a 240 vs 250, like what happens in reality.
Click to expand...

I wonder how that interval is even estimated when nobody is allowed to take the test twice.
 
efle said:
I mean hell, the NBME themselves had the greatest stake in keeping this exam scored. They make a BOATLOAD off selling their practice exam forms. Even they, after seeing all sorts of internal data that we are not privy to, had to do it.

Mark my words, they didn't want this either when they launched the INCUS ordeal last year. There's a lot behind the scenes (like the random sample survey results, or their internal data on test form performance/score creep) that made them feel this was necessary.
Click to expand...

So just to be clear, your argument, without knowledge of this or having seen a shred of actual data, is that the NBME is being benevolent with this? That the same people who gave us Step 2 CS are only looking out for the medical students best interests...

If they have internal data driving this they are obligated to release it. If we don’t see any published data by the end of this year then there should be a very high suspicion of an agenda, even higher than currently exists.
 
efle said:
Totally agree. Though, I can't imagine any agenda that would drive the NBME to do this if it was unpopular. They only gain from Step 1 Mania.
Click to expand...

The AAMC is their biggest stakeholder. We’re still paying them to take a Pass/Fail Step 1 regardless.
 
anyone stating the previous score had to high of a variance - why didnt they just attempt to lower the varience of the score, make scoring tighter, or anything else rather than turning it pass/fail. logic doesnt add up
 
ace_inhibitor111 said:
I wonder how that interval is even estimated when nobody is allowed to take the test twice.
Click to expand...
There's a good deal of statistical tricks out there. For example, randomly take half the items I answered and compare to the other random half of items. If I got exactly 135/140 on both halves, put me down as 100% agreement. Do this for a thousand test-takers and build a bell curve of the discrepancy.
 
efle said:
There is no way to write a knowledge-check test across 2 years of content with a narrow interval. The only way tests like the SAT or MCAT retain tight intervals is by testing aptitude with novel content instead of primarily going after knowledge base. When they only get 1 or 2 questions to assess the strength of your knowledge base for entire categories, there's no way to avoid having a 30+ point interval for a good p value.

My personal p=.05 interval spanned from 230s to 270s. That is insane. This test was never designed to do head-to-head comparisons of applicants with a 240 vs 250, like what happens in reality.
Click to expand...

increase the length of the exam or make it multiple parts. Or change the style of the exam.

Do something that retains a student's ability to have upward mobility.
 
AnatomyGrey12 said:
I predict a number of schools return to a fully graded and ranked pre-clinical system.



Go take a gander at who the loudest and biggest proponents were of a P/F USMLE Step 1.... Almost universally the push came from individuals associated with large, highly ranked, and well known institutions.
Click to expand...
Medical students supported the change also.
AMSA endorsed going to pass fail
I remember several threads on SDN endorsing the idea of pass fail Step.
Well, the game.has now changed. Students will now have to decide how to play the game well. It's the only game in town.
Students lobbying NBME should have presented a more comprehensive plan rather than just going to pass fail and letting NBME figure it out. There are always unintended consequences. I feel the DO students now have lost an important tool in distinguishing themselves. Rotation evals are very subjective. Now students will be stressed as they will be putting in long days on rotations AND studying for CK.Lastly, There is little time and less data on how to play the NEW Match Game. I. wish I had something positive to offer
 
AnatomyGrey12 said:
So just to be clear, your argument, without knowledge of this or having seen a shred of actual data, is that the NBME is being benevolent with this? That the same people who gave us Step 2 CS are only looking out for the medical students best interests...

If they have internal data driving this they are obligated to release it. If we don’t see any published data by the end of this year then there should be a very high suspicion of an agenda, even higher than currently exists.
Click to expand...
My argument is that the NBME only stands to benefit, ergo if they are announcing it Pass/Fail (I was expecting only Quartiles), they sure as hell aren't doing it because of a 50-50 split of public comments. The way they behaved about Step 2 CS makes my point, they don't do something because it is right. They do what is profitable or what is necessary, and this certainly won't be a profitable change. Like I said though, completely agree about seeing data.
 
efle said:
There's a good deal of statistical tricks out there. For example, randomly take half the items I answered and compare to the other random half of items. If I got exactly 135/140 on both halves, put me down as 100% agreement. Do this for a thousand test-takers and build a bell curve of the discrepancy.
Click to expand...

Hold on, that doesn’t make sense. The proportion you got wrong is a fixed percentage. The probability of getting equal halves of a test correct assuming random assortment only depends on the number of questions.
 
Lawper said:
why not
Click to expand...

Because specialty exams test on basic knowledge are generally much easier than the Steps. Asking them to do a med student version would result in the IM exam asking what CHF is.
 
Mass Effect said:
Yeah, I don't give much credence to surveys given before score release.
Click to expand...
To the contrary, they wanted to know things like how it impacted your mental health. Not gonna get an unbiased answer after someone finds out they killed it or bombed it. They'd get nothing but "keep it scored!" from the high end and "make it Pass" from the low end, if they waited until after.
 
ace_inhibitor111 said:
Hold on, that doesn’t make sense. The proportion you got wrong is a fixed percentage. The probability of getting equal halves of a test correct assuming random assortment only depends on the number of questions.
Click to expand...
Correct, the only way they could shrink their interval would be to make it into a multi-day test with half a thousand or more questions. And even then, the rate of interval shrink from adding items is not great.
 
@efle I honestly feel like the necessity was institutional. There is no way that medical schools could keep up to the quality of services like SketchyMedical or even have similar quality lecturers to professors like Goljan, Najeeb, Ryan, Sattar, or Williams. Students have been more recently vocal about noting how much they pay for medical school compared to how little they pay for any of these online lecture services. The fact that there are memes about wellness lectures and mandatory lectures scheduled right before big block exams compared to ones glorifying services like uWorld e.g. "What did people even do before uWorld?" indicated to me that schools were under increased scrutiny especially with the primary political campaign platforms being on student loans and tuition costs.

Some adcoms have been direct that the tuition model is really appropriated for schools to fund rotations in the third year and the amount of tuition students spend for M1 and M2 is really to fund their own clinical clerkships down the road. However, the idea that students are basically paying into schools that simply raise more barriers to access education rather than reducing them is what I saw as being a large pressure to turn Step 1 into P/F in order for schools to be "let off the hook" from the mounting pressure as they were being compared to free market services that were doing their job much better for a much cheaper price. I think that a lot of student input is negligible to the overall landscape. Polls are hardly exhaustive and are a poor metric in which to pivot, but when institutional pressure is to encourage a P/F system from prestigious to "lower tier" schools then there is no reason that the changes would no go through at some point in time.
 
efle said:
Correct, the only way they could shrink their interval would be to make it into a multi-day test with half a thousand or more questions. And even then, the rate of interval shrink from adding items is not great.
Click to expand...

But what I’m saying is they can’t use that method since it doesn’t really affect anything. The MCAT had 230 questions and apparently a tighter interval. The only thing I can think off is the NBME estimated the variance by comparing the answers of say a 260 scorer with that of a 220 scorer, and seeing if all higher scorers got that same question correct vs all the low scorers, etc. But this would overestimate the actual variance when testing a very large body of knowledge.
 
efle said:
To the contrary, they wanted to know things like how it impacted your mental health. Not gonna get an unbiased answer after someone finds out they killed it or bombed it. They'd get nothing but "keep it scored!" from the high end and "make it Pass" from the low end, if they waited until after.
Click to expand...

But everyone thinks they failed before score release.
 
libertyyne said:
ya'll are doomed. jk its status quo unless residencies start testing changes before hand.
Click to expand...

I saw someones comment saying its possible we might get a 3 digit score when we take the test this summer, then when we apply to residency it could just be listed as a 'P', is that accurate?
 
Lawper said:
so make it a lot harder for stratifying?
Click to expand...

And who monitors how hard it is, the NBME? LOL. The fact is that until you're in the specialty, you shouldn't be expected to know the nitpicky nuances of the field. At the same time, if you want to stratify students, it needs to be somewhat difficult. The specialty boards aren't the ones to strike that balance and the NBME is not going to forfeit the task to them.
 
You must log in or register to reply here.

Similar threads

cerebellar
Step 1 advice!
Replies
0
Views
458
cerebellar
cerebellar
E
Uworld step 1 qbank
Replies
0
Views
44
ellebelle_8
E
S
M1 Step 1 Advice
Replies
4
Views
468
Goro
Goro
C
the state of step 1
2
Replies
89
Views
8K
DO2015CA
DO2015CA
Sram.jpg
Changing From P/F to Letter Grades
Replies
7
Views
1K
14_karat
14_karat
Top