Step 1 P/F: Decision

[libertyyne]

---

Members don't see this ad.

Haven’t seen this discussed much.. What about schools that have P/F preclinicals vs grades? Will students with good grades have an advantage over students from another school with a pass for pre-clinicals?

Most schools with true P/F are higher up the food chain anyway. Others may claim p/f but a really ranking you internally anyway. Plus school grades are kind of useless how does one even compare two schools ? especially in a world where the datasets are small.

 

[libertyyne]

I’m really frustrated after reading the updated website earlier today. The NBME has literally none of this figured out and wants to implement a drastic change in two years. Thousands of people in the M1 class don’t know what to prioritize in terms of Step studying because we don’t know how Step 1 will be scored for our class, and the leadership just says “hang on, we’ll update you several months from now with more information”?

you should be prioritizing learning the stuff anyway .It will help in learning the CK stuff. Keep on grinding.

 

[sylvanthus]

And the rising cost of healthcare continues. You shouldn't need a medicine consult for bread and butter medical knowledge.

Agreed, but lets be real, surgery calls medicine all the time for glucose management, blood pressure management, etc and ive seen mult times the surgeon request medicine as primary during admission so they can "medically optimize" the pt. It appears either basic medical knowledge is lost on some surgeons, or they are attempting to turf. It happens more than people admit.

 

[subdermallight]

Most schools with true P/F are higher up the food chain anyway. Others may claim p/f but a really ranking you internally anyway. Plus school grades are kind of useless how does one even compare two schools ? especially in a world where the datasets are small.

What about schools that internally rank but only report the top x% of the class in the dean letter? I’ve always wondered if RDs will all know you were not in the top because they don’t see that line on letter while some other students had it. Or if they don’t really look into it that much

 

[libertyyne]

What about schools that internally rank but only report the top x% of the class in the dean letter? I’ve always wondered if RDs will all know you were not in the top because they don’t see that line on letter while some other students had it. Or if they don’t really look into it that much

maybe they start paying more attention, Any strategy premised on PD ignorance is not a good one. If you want to be perceived as the top of your class you should work to get there.

 

[Phloston]

I’m really frustrated after reading the updated website earlier today. The NBME has literally none of this figured out and wants to implement a drastic change in two years. Thousands of people in the M1 class don’t know what to prioritize in terms of Step studying because we don’t know how Step 1 will be scored for our class, and the leadership just says “hang on, we’ll update you several months from now with more information”?

Study for Step 1 as though nothing has changed. And if you do get shafted and need to take it as P/F, your hard work will make your 2CK higher. I studied super-hard for Step 1, and probably ~50% effort for Step 2, and scored the same on both. Step 1 will prepare you for 2CK. So if you gun Step 1 then also gun Step 2 (cuz you'll have to), you'll kill it.

 

[Phloston]

But I'm sure almost all programs will require step 2 scores now.

Not almost all. All.

 

[Phloston]

holy cow this is peak incompetence. I actually do think that going P/F is the right move long term but it’s so clear this was a hastily, poorly thought out decision that is going to tick a lot of people off. A plan or decision should already have been in place with the announcement and they are in a lose-lose situation no matter what the decision is.

I think their lack of definitive answer is simply because they're aware they didn't think it through. Clearly a retroactive cover-up would be worse than the mess they've already created.

 

[efle]

Study for Step 1 as though nothing has changed. And if you do get shafted and need to take it as P/F, your hard work will make your 2CK higher. I studied super-hard for Step 1, and probably ~50% effort for Step 2, and scored the same on both. Step 1 will prepare you for 2CK. So if you gun Step 1 then also gun Step 2 (cuz you'll have to), you'll kill it.

I mean, that might be more due to the free +15 points higher average and the fact that nobody else tried nearly as hard on Step 2 either...

 

[Mad Jack]

No one was talking about medically complex patients when it's not straight forward. The point is that the majority of what you see IS straight forward and I don't know about you, but I sure wouldn't want to be on the consulting end of calling cards to tell them you have a patient with BP 140/70.



Outpatient medicine is also relevant to surgery during training anyway. If you have an outpatient clinic, understanding what's going on with your patients medically is important. As a psychiatrist, I comment on laboratory values, I read my own EKGs, I sometimes evaluate my own imaging (there's always a read, obviously, but I look at it too), I send notes to PCPs if I start a patient on a psych med and tell them I'll be ordering LFTs or what to look out for in terms of side effects during their next visit, I may start propranolol for akathisia, etc etc. Basic medical knowledge is relevant to every field of medicine.

And yet I get at least two to three consults a day for "history of depression/anxiety" that is stable on their home regimen

 

[Mass Effect]

And yet I get at least two to three consults a day for "history of depression/anxiety" that is stable on their home regimen

Me too, which speaks to the importance of other specialties knowing the basics of medicine.

 

[arrianym]

So if you were still in college and knew Step 1 would be pass/fail and you didn't get into a top med school, you wouldn't have chosen med school at all? Come on, you're not guaranteed a 260 on Step 1. You can study your ass off and still get slammed. Just ask in the Step forums. Everyone thinks they'll get that coveted score and everyone is quick to say they did, but the reality is that most people score average (which is why it's the average). SDN is the only place on Earth where 99% of the population scored over 250. Everyone needs to take a breath and realize that when the metrics change, you change your strategy. Shouting about the unfairness of it is counter-productive.



It was a stupid equalizer. It had little to do with how you'd perform in residency and was never meant to be any kind of equalizer.

Honestly, I would still apply. BUT I would feel so much more pressure to get into a "prestigious" school. I worked my BUTT off in college, and I did pretty well. However, the applicant pool is really tough and I currently go to a lower tier MD school. It's a great fit and I love it here, and I know that I'm not limited by my school's rank when it comes to getting a good residency. There are students in my class who turned down higher ranked schools because this was a better fit. However, after this policy change, finding a good fit might be less of a priority if you know that the school's prestige factors into your residency opportunities.

And by the way, I'm not saying that STEP is perfect. I'm currently a first year and I have no idea how I'll do. HOWEVER, making STEP pass/fail does not provide a solution to residencies, which cannot possibly read though the thousands of applications they are bombarded with. Residencies will need to find another way to filter through the applications. So now, medical students have additional pressure to boost their research, EC's, etc...Not that we're not already doing that, but at least we know that education is our first priority and checking these other boxes like we had to as pre-meds isn't as vital. So instead of stressing about STEP, students will need to stress about EC's, research, letters of rec even more! And that's on top of their already rigorous courses. AND, students that go to "prestigious" schools will definitely have an upper hand. I suppose if STEP 2 became the new STEP 1, this policy might be for the better. However, the timing is pretty stressful. But like everyone, I'm not completely sure what's going to happen. What I do know though, is that mocking or invalidating people's stress isn't right, because this kind of thing affects an important point in their career...that's all I was saying

 

[ooh a bean]

Honestly, I would still apply. BUT I would feel so much more pressure to get into a "prestigious" school. I worked my BUTT off in college, and I did pretty well. However, the applicant pool is really tough and I currently go to a lower tier MD school. It's a great fit and I love it here, and I know that I'm not limited by my school's rank when it comes to getting a good residency. There are students in my class who turned down higher ranked schools because this was a better fit. However, after this policy change, finding a good fit might be less of a priority if you know that the school's prestige factors into your residency opportunities.

And by the way, I'm not saying that STEP is perfect. I'm currently a first year and I have no idea how I'll do. HOWEVER, making STEP pass/fail does not provide a solution to residencies, which cannot possibly read though the thousands of applications they are bombarded with. Residencies will need to find another way to filter through the applications. So now, medical students have additional pressure to boost their research, EC's, etc...Not that we're not already doing that, but at least we know that education is our first priority and checking these other boxes like we had to as pre-meds isn't as vital. So instead of stressing about STEP, students will need to stress about EC's, research, letters of rec even more! And that's on top of their already rigorous courses. AND, students that go to "prestigious" schools will definitely have an upper hand. I suppose if STEP 2 became the new STEP 1, this policy might be for the better. However, the timing is pretty stressful. But like everyone, I'm not completely sure what's going to happen. What I do know though, is that mocking or invalidating people's stress isn't right, because this kind of thing affects an important point in their career...that's all I was saying

I always found it ironic that the people responding this way to people's stress are almost all residents or attendings. So much for needing to have empathy in order to become a doctor.

 

[Alevy123]

I think the key statement from the announcement is: “No EARLIER than Jan 2022” which means they could make the change go into effect after the c/o of 2023 has matched. That way you don’t have the issue of a mix of applicants with 3 digit scores and applicants with p/f

 

[slowthai]

I think the key statement from the announcement is: “No EARLIER than Jan 2022” which means they could make the change go into effect after the c/o of 2023 has matched. That way you don’t have the issue of a mix of applicants with 3 digit scores and applicants with p/f

If you go to the NBME's website, they have an FAQ section where they said they don't know if they're going to retroactively change scores for the people taking it in 2021. Furthermore, the match for the class of 2023 occurs in March 2023.

 

[Matthew9Thirtyfive]

If you go to the NBME's website, they have an FAQ section where they said they don't know if they're going to retroactively change scores for the people taking it in 2021. Furthermore, the match for the class of 2023 occurs in March 2023.

Yeah but apps open September 2022.

 

[subdermallight]

Does anyone know if the statistical error on Step 2 is just as bad as Step 1 in terms of scoring?

 

[slowthai]

Yeah but apps open September 2022.

But they could still decide to remove your score, based on what they said.

 

[Matthew9Thirtyfive]

But they could still decide to remove your score, based on what they said.

I mean if you submit September 15, and programs download your application October 1, I’m not sure how they would remove your score. Are they going to go to each program and white out all the scores?

 

[slowthai]

I mean if you submit September 15, and programs download your application October 1, I’m not sure how they would remove your score. Are they going to go to each program and white out all the scores?

But that's what I'm saying. Let's say I take it in July 2021. I would be applying for the match in September of 2022. The policy goes into effect in January of 2022. That means that before I'm even able to apply, they could decide to remove my score.

 

[Matthew9Thirtyfive]

But that's what I'm saying. Let's say I take it in July 2021. I would be applying for the match in September of 2022. The policy goes into effect in January of 2022. That means that before I'm able to even apply, they could decide to remove my score.

Oh yeah. I was referring to the idea that maybe they would do it in 2023.

 

[slowthai]

Oh yeah. I was referring to the idea that maybe they would do it in 2023.

Ohhh I even misinterpreted the person I quoted, lol. You're totally right. I really hope that's the case. Hopefully it happens like it did in dentistry where they keep delaying it for a few years.

 

[libertyyne]

Does anyone know if the statistical error on Step 2 is just as bad as Step 1 in terms of scoring?

Yes, the SEM is similar and the SD is similar. It is just as bad as step 1 in that regard. Better because its more clinically focused, worse because it is later.

 

[Mass Effect]

What I do know though, is that mocking or invalidating people's stress isn't right, because this kind of thing affects an important point in their career...that's all I was saying

I haven't seen anyone mock you or anyone else. I have seen people, including myself, criticize some of the reactions because some of them are truly over the top. One poster said this means that anyone with a pencil and a pulse can now become a doctor. You don't think that's the epitome of melodrama? Our responses come from those types of posts, not from people who lay out legitimate concerns because we get that this is a change and there will be some adjusting of expectations, attitudes, and figuring out the formula to success. So yeah, for the first class or two that goes through this change, it will be anxiety-provoking, understandably so. But overall, this is a good change and you'll realize that when you get to the other side and see how very little Step 1 has to do with anything remotely important in medicine. That's all anyone's trying to say.

 

[Splenda88]

Schools might have to change their curriculum to 3 years so students can take step 1/2 back to back. It would be a good thing for all parties involved except for the schools since they will collect less tuition.

 

[ciestar]

I always found it ironic that the people responding this way to people's stress are almost all residents or attendings. So much for needing to have empathy in order to become a doctor.

For some, it is the “i had to deal with it (suffer) so you have to, too” mentality.

 

[NotAProgDirector]

Does anyone know if the statistical error on Step 2 is just as bad as Step 1 in terms of scoring?

Yes, the SEM is similar and the SD is similar. It is just as bad as step 1 in that regard. Better because its more clinically focused, worse because it is later.

I've mentioned this before, but the statistical error on S1 isn't really bad. You're misinterprting the meaning of SEM. Plus, when you use +/- 2 SD's, you're setting a 95% confidence interval. Nothing else I use has that level of accuracy.

 

[ExcitatorySynapse]

Not gonna lie, I was kind of halfhearted about step studying (MD/PhD-MS1) so far and I don’t know how much I should invest into it now, given that the very earliest I’ll apply for match is September 2025. Physiology is dull and I’d rather study material relevant to my research. Any thoughts?

 

[Splenda88]

Not gonna lie, I was kind of halfhearted about step studying (MD/PhD-MS1) so far and I don’t know how much I should invest into it now, given that the very earliest I’ll apply for match is September 2025. Physiology is dull and I’d rather study material relevant to my research. Any thoughts?

Wait until you take histology...

 

[FindersFee5]

I've mentioned this before, but the statistical error on S1 isn't really bad. You're misinterprting the meaning of SEM. Plus, when you use +/- 2 SD's, you're setting a 95% confidence interval. Nothing else I use has that level of accuracy.

THANK YOU. This is the thing I keep coming back to with this - there are almost no accurate measures of comparison that will be able to predict the outcome of a single event between two participants with 95% confidence. Statistical significance for scientific research is intentionally set much higher than most real life events. I think a good comparison is Elo rating in chess - you need to have a 200 point difference before one person is predicted to win even 75% of matches (less than the 95% confidence interval you commonly see people using to discredit Step). Yet a player who is 200 points higher than you on the Elo scale is a much better player.

By the way, @efle - you've said a few times that the MCAT is a better test because it's measuring fixed traits so the standard deviation is smaller. That's just not true. The standard deviation on the MCAT is 10.6 points. That's HORRENDOUS. If we're applying the same 95% confidence interval, that means an average medical school applicant (505.6) cannot be differentiated from an incredible applicant (527) and somebody who's app would be DOA (484).

 

[efle]

I've mentioned this before, but the statistical error on S1 isn't really bad. You're misinterprting the meaning of SEM. Plus, when you use +/- 2 SD's, you're setting a 95% confidence interval. Nothing else I use has that level of accuracy.

...and that 95% interval is an insane >30 points wide. For many people a 16-point swing in their score would drastically change their competitiveness, and that is literally the difference between hitting the lucky versus unlucky bounds of their 65% CI.

 

[efle]

THANK YOU. This is the thing I keep coming back to with this - there are almost no accurate measures of comparison that will be able to predict the outcome of a single event between two participants with 95% confidence. Statistical significance for scientific research is intentionally set much higher than most real life events. I think a good comparison is Elo rating in chess - you need to have a 200 point difference before one person is predicted to win even 75% of matches (less than the 95% confidence interval you commonly see people using to discredit Step). Yet a player who is 200 points higher than you on the Elo scale is a much better player.

By the way, @efle - you've said a few times that the MCAT is a better test because it's measuring fixed traits so the standard deviation is smaller. That's just not true. The standard deviation on the MCAT is 10.6 points. That's HORRENDOUS. If we're applying the same 95% confidence interval, that means an average medical school applicant (505.6) cannot be differentiated from an incredible applicant (527) and somebody who's app would be DOA (484).

You're confusing standard dev with retest error measures. The retest for MCAT for the same CI was +/- 2 pts on the scale I took it on (dunno if it's the same on the 500 scale)

which is exactly making my point, by the way. Two populations, both with massive population Std.devs, but one test with a 400% worse interval for placing an individual in that curve.

 

[FindersFee5]

You're confusing standard dev with retest error measures. The retest for MCAT for the same CI was +/- 2 pts on the scale I took it on (dunno if it's the same on the 500 scale)

which is exactly making my point, by the way. Two populations, both with massive population Std.devs, but one test with a 400% worse interval for placing an individual in that curve.

Where are you finding the retest error for step 1 and MCAT?

 

[efle]

Where are you finding the retest error for step 1 and MCAT?

Both I believe are on the score sheets given to individuals after their test. Both I believe are also in the USMLE and AAMC pdfs about their exams. I can find for you later if you can't find yourself

Edit: USMLE - SED 8 pts for Step 1

https://www.usmle.org/pdfs/transcripts/USMLE_Step_Examination_Score_Interpretation_Guidelines.pdf

Edit: MCAT - The reliability of the total score is .95, and the standard error of measurement for the total score is 2. Section score reliabilities range from .82 to .86, and the standard errors of measurement for section scores are 1.

https://www.aamc.org/system/files/c/2/454206-mcatatglance2015.pdf

So no, someone isn't really going to luck their way into a 525 instead of a 515, whereas on the USMLE it is completely possible for luck at the 65% level to swing you between 235 to 250 if you lie near the center.

 

[efle]

Does present an interesting thought experiment, though. If the MCAT did have a garbage interval that could barely distinguish a top decile applicant from someone at risk of needing extra years/board repeats...would there still be a vocal group of high scorers defending it's role in medical admissions?

Probably.

 

[FindersFee5]

Both I believe are on the score sheets given to individuals after their test. Both I believe are also in the USMLE and AAMC pdfs about their exams. I can find for you later if you can't find yourself

Edit: USMLE - SED 8 pts for Step 1

https://www.usmle.org/pdfs/transcripts/USMLE_Step_Examination_Score_Interpretation_Guidelines.pdf

Edit: MCAT - The reliability of the total score is .95, and the standard error of measurement for the total score is 2. Section score reliabilities range from .82 to .86, and the standard errors of measurement for section scores are 1.

https://www.aamc.org/system/files/c/2/454206-mcatatglance2015.pdf

So no, someone isn't really going to luck their way into a 525 instead of a 515, whereas on the USMLE it is completely possible for luck at the 65% level to swing you between 235 to 250 if you lie near the center.

I'd argue that a 508 to a 512 would swing a lot of people's applications, but otherwise agreed - your point is well taken. 235 to 250 would be absolutely life changing for people gunning for competitive specialties.

The argument for step 1 is actually stronger if you look across all specialties. I'd guess that the range admins care about for MCATs is probably about 506 - 520 - scores above and below those are kind of lumped together. For step 1, maybe 210 - 260, if you're comparing across all specialties? So that range/standard error of estimate is about the same for both (7 vs 6.25). The problem is that many competitive specialties effectively decrease that range to around 235 - 260, which is where you get those huge outcome swings from retest error.

 

[efle]

I'd argue that a 508 to a 512 would swing a lot of people's applications, but otherwise agreed - your point is well taken. 235 to 250 would be absolutely life changing for people gunning for competitive specialties.

The argument for step 1 is actually stronger if you look across all specialties. I'd guess that the range admins care about for MCATs is probably about 506 - 520 - scores above and below those are kind of lumped together. For step 1, maybe 210 - 260, if you're comparing across all specialties? So that range/standard error of estimate is about the same for both (7 vs 6.25). The problem is that many competitive specialties effectively decrease that range to around 235 - 260, which is where you get those huge outcome swings from retest error.

The range that people care about isn't factored into the test design, though. The overall bell is. And it gets worse when you think more about where they're forced to set their lower end versus average.

For USMLE, they have to set their minimum passing score (~5th percentile) at 65-70% correct. This essentially forces them to stuff 95% of test takers into being differentiated by a much smaller set of questions than they otherwise could use. Compare that to the MCAT, where they could set 5th percentile at 15% correct, median at 50% correct, and 95th percentile at 85% correct if they so chose. Much more room to play with.

It's just an ugly, sloppy, messy, totally abused metric.

 

[efle]

Specifically, here is what the old MCAT used to look like in terms of percentage correct against percentile. You can see they're making use of a HUGE region, roughly 30-100% correct.

Efle's Percent Correct vs Percentile Earned Charts for the Old MCAT

I was always curious about the relationship between number of correct answers and resulting score percentiles. So I decided to go ahead and plot it out when I saw someone asking about it earlier in a thread. The charts are built using the scoring scales for AAMC Official Practice Tests 9, 10...

forums.studentdoctor.net

USMLE having ~69% correct as the 5th percentile is a HUGE barrier to duplicating that kind of function.

 

[FindersFee5]

Specifically, here is what the old MCAT used to look like in terms of percentage correct against percentile. You can see they're making use of a HUGE region, roughly 30-100% correct.

Efle's Percent Correct vs Percentile Earned Charts for the Old MCAT

I was always curious about the relationship between number of correct answers and resulting score percentiles. So I decided to go ahead and plot it out when I saw someone asking about it earlier in a thread. The charts are built using the scoring scales for AAMC Official Practice Tests 9, 10...

forums.studentdoctor.net

USMLE having ~69% correct as the 5th percentile is a HUGE barrier to duplicating that kind of function.

That is interesting - so the differentiation is essentially ~95/152 questions for old MCAT and ~70/280 for step 1.

I wonder if the real solution would have been to increase number of questions on step 1 across 2 days of testing? 480 questions spread across 2 days would give you significantly more differentiation ability even while retaining the 69% 5th percentile.

 

[Lawpy]

Specifically, here is what the old MCAT used to look like in terms of percentage correct against percentile. You can see they're making use of a HUGE region, roughly 30-100% correct.

Efle's Percent Correct vs Percentile Earned Charts for the Old MCAT

I was always curious about the relationship between number of correct answers and resulting score percentiles. So I decided to go ahead and plot it out when I saw someone asking about it earlier in a thread. The charts are built using the scoring scales for AAMC Official Practice Tests 9, 10...

forums.studentdoctor.net

USMLE having ~69% correct as the 5th percentile is a HUGE barrier to duplicating that kind of function.

The classic @efle conversion charts have returned

 

[subdermallight]

Thank you @efle for doing all the math. It’s too bad you and @FindersFee5 weren’t in charge of this decision

 

[deleted564680]

Conspiracy theory:

USMLE Step 1 made to be P/F so NPs could take it in the coming years, "pass" it without anyone knowing their real scores, and thus obtain "true parity" with physicians.

 

[efle]

Thank you @efle for doing all the math. It’s too bad you and @FindersFee5 weren’t in charge of this decision

I would've also made it Pass/Fail haha

 

[Thesimplelifeofamyloid]

I would've also made it Pass/Fail haha

Or they could have just fixed the exam and the statistical issue with it. People act like an exam is unfixable without even proposing solution that takes all parties into account.. Part of me thinks you hate this exam so much because it didn't tell you that you were at the ~99 percentile of test takers

 

[efle]

Or they could have just fixed the exam and the statistical issue with it. People act like an exam is unfixable without even proposing solution that takes all parties into account.. Part of me thinks you hate this exam so much because it didn't tell you that you were at the ~99 percentile of test takers

It wasn't designed to perform this function, so the fix it'd need would be a replacement with something entirely different. That's just as easy to do now. Filling that need is probably easier now, actually.

You can find plenty of posts in my history criticizing Step from before I had my score. Besides, I wasn't the zanki-maturing 270 or bust type, I was the >250 to keep all doors open type, and I managed it. I really don't have a chip on my shoulder about 230s being treated differently than 250s, if anything that benefits me. I really, truly found the criticisms a lot more convincing (especially as presented by Carmody) than any of the defenses.

 

[ace_inhibitor111]

Conspiracy theory:

USMLE Step 1 made to be P/F so NPs could take it in the coming years, "pass" it without anyone knowing their real scores, and thus obtain "true parity" with physicians.

Uggh.. I have no doubt a good portion of NPs can pass this test. The threshold is more like 60% correct, not 69%. They should just get rid of Step 1 entirely. Passing it doesn’t have any value, it doesn’t tell you if you’re able to understand basic science or are a safe physician. Now it’s just there to make money like CS.

 

[Thesimplelifeofamyloid]

It wasn't designed to perform this function, so the fix it'd need would be a replacement with something entirely different. That's just as easy to do now. Filling that need is probably easier now, actually.

You can find plenty of posts in my history criticizing Step from before I had my score. Besides, I wasn't the zanki-maturing 270 or bust type, I was the >250 to keep all doors open type, and I managed it. I really don't have a chip on my shoulder about 230s being treated differently than 250s, if anything that benefits me. I really, truly found the criticisms a lot more convincing (especially as presented by Carmody) than any of the defenses.

Fair enough. Hopefully what I was saying wasn't coming off too antagonistic

 

[ace_inhibitor111]

It wasn't designed to perform this function, so the fix it'd need would be a replacement with something entirely different. That's just as easy to do now. Filling that need is probably easier now, actually.

You can find plenty of posts in my history criticizing Step from before I had my score. Besides, I wasn't the zanki-maturing 270 or bust type, I was the >250 to keep all doors open type, and I managed it. I really don't have a chip on my shoulder about 230s being treated differently than 250s, if anything that benefits me. I really, truly found the criticisms a lot more convincing (especially as presented by Carmody) than any of the defenses.

I would argue there isn’t a simple way to replace Step 1. You seem to want to replace it with a psychometric, WAIS-III like test, but that will never happen. Not with the current emphasis on holistic selection. Either way, we don’t need people with 160 IQs in Derm or Ortho. The good thing about Step 1 was it didn’t just test IQ. It also tested hard work, willpower and overall memory capacity, which are important qualities of residents.

I disagree with Carmody’s argument against the Step 1. First of all, it’s not like he cited available research. Most of the data suggest a mild-moderate association between Step 1 score and residency success. The issue is that residency evaluations are mostly subjective, so correlating an objective score to subjective evaluations will always yield modest results. There is a greater likelihood that the NBME starts scoring CS rather than having a more objective test. At that point the next generation of students will all be screwed.

 

[efle]

I would argue there isn’t a simple way to replace Step 1. You seem to want to replace it with a psychometric, WAIS-III like test, but that will never happen. Not with the current emphasis on holistic selection. Either way, we don’t need people with 160 IQs in Derm or Ortho. The good thing about Step 1 was it didn’t just test IQ. It also tested hard work, willpower and overall memory capacity, which are important qualities of residents.

I disagree with Carmody’s argument against the Step 1. First of all, it’s not like he cited available research. Most of the data suggest a mild-moderate association between Step 1 score and residency success. The issue is that residency evaluations are mostly subjective, so correlating an objective score to subjective evaluations will always yield modest results. There is a greater likelihood that the NBME starts scoring CS rather than having a more objective test. At that point the next generation of students will all be screwed.

Oh no, don't get me wrong. I think if PDs are going to boil people down to a number, it's best for it to be a number that reflects clinical reasoning more than Step did. But I'm not a fan of boiling people down to a number at all. Like I keep saying, I think even the system of the 1990s with no such number was better than what it's become.

Did...did you read his blog? Every occasion he can he goes for the data. The only thing that had any substance was the oft cited correlation of Step to board passage rates. But, that effect was down at the lower end (e.g. a 210 is safer than a 195) with no tangible gains from all the favoritism of the other side of the curve.

The one big issue I don't like is that, like you and others have said, this is going to rob a lot of outlier IMG/DO candidates from identifying themselves. But claiming the way PDs and students have been chasing the high end has data backing up moderately better residency success? I call shenanigans.

Uggh.. I have no doubt a good portion of NPs can pass this test. The threshold is more like 60% correct, not 69%. They should just get rid of Step 1 entirely. Passing it doesn’t have any value, it doesn’t tell you if you’re able to understand basic science or are a safe physician. Now it’s just there to make money like CS.

Varies from 60-70% depending on form per their PDFs and, due to score creep/constantly adjusting upwards, would expect it to be at the upper side of that range now on average. Something like, oh I dunno, 194/280 post-scaling between forms (the current Pass cutoff) !

 

[ace_inhibitor111]

Oh no, don't get me wrong. I think if PDs are going to boil people down to a number, it's best for it to be a number that reflects clinical reasoning more than Step did. But I'm not a fan of boiling people down to a number at all. Like I keep saying, I think even the system of the 1990s with no such number was better than what it's become.

Did...did you read his blog? Every occasion he can he goes for the data. The only thing that had any substance was the oft cited correlation of Step to board passage rates. But, that effect was down at the lower end (e.g. a 210 is safer than a 195) with no tangible gains from all the favoritism of the other side of the curve.

The one big issue I don't like is that, like you and others have said, this is going to rob a lot of outlier IMG/DO candidates from identifying themselves. But claiming the way PDs and students have been chasing the high end has data backing up moderately better residency success? I call shenanigans.


Varies from 60-70% depending on form per their PDFs and, due to score creep/constantly adjusting upwards, would expect it to be at the upper side of that range now on average. Something like, oh I dunno, 194/280 post-scaling between forms (the current Pass cutoff) !

Link me to where he cites research on Step 1. His blog from what I saw consists of rehashed NRMP Statistics and that one post where he cherry-picked Step 1 questions to show how ridiculous it was but IMO were still quite valid.

The new NBMEs consistently show 60% correct as passing, and they’re not generally as hard as the real exam.

Reddit.com/r/step1/comments/bjmd91

You can check the Step 1 research yourself. I’ll post random links but a literature search would probably do more good.

The Predictive Value of General Surgery Application Data for Future Resident Performance

The predictive value of application data for future general surgery resident performance and attrition are poorly understood. We sought to determine w…

www.sciencedirect.com

What predicts surgical internship performance?

Variables associated with postgraduate year 1 (PGY-l) performance in surgical training have not been fully defined.Mean composite PGY-l evaluation sco…

www.sciencedirect.com

United States Medical Licensing Examination Step 1 Scores Directly Correlate with American Board of Neurological Surgery Scores: A Single-Institution Experience

Neurosurgery residency is becoming an increasingly competitive match. The process of screening and ranking applicants is a multifactorial process that…

www.sciencedirect.com

Correlation of USMLE Step 1 scores with performance on dermatology in-training examinations

Although United States Medical Licensing Examination (USMLE) Step 1 was not designed to predict resident performance, scores are used to compare resid…

www.sciencedirect.com

https://www.researchgate.net/publication/266205167_Usmle_Step_1_Score_a_Valid_predictor_of_Success_in_Surgical_Residency