Pass/ Fail Step 1

This forum made possible through the generous support of SDN members, donors, and sponsors. Thank you.
Status
Not open for further replies.
Another issue with making the app "more holistic" is that is really code for pre-med style volunteer in a soup kitchen to show I'm Mother Theresa type bull****. No thanks. I didn't do it for med school apps and I'm definitely not doing that for residency apps. In fact, I would blackball anyone who is in student government or anything like that but that's my bias showing the other extreme view of that situation. If by holistic they mean spends time with family, does hobbies and isn't a weirdo then yeah we already have that and don't need pass/fail to observe that.
Members don't see this ad.
 
  • Like
Reactions: 16 users
chlorocruorin said:
I used to believe this, until I saw a few physicians discussing this over social media. Although those are the easiest ways to measure applicants, is it not up to residencies to come up with a correct method of stratifying applicants that would not have our fates sealed before entering medical school? There are pros to P/F which is why it has been proposed. At what point will residencies say we need to stop looking at step 1 scores to stratify applicants. Will my grandchildren need 280 to match into ortho and be deemed a "successful medical student".
Click to expand...

So what metrics are the alternative? Do start paying for specialty specific exams on top of everything else? Do we sit down and take 4 hour exams before each residency interview? Or do we accept that standardized testing is the norm and just grade step 1. Some honestly act like we haven't had to take a graded standardized test before and it's barbaric that it determines our future. The SAT, ACT, GRE, MCAT all exist. We've all taken them.
 
  • Like
  • Love
Reactions: 5 users
Neopolymath said:
Another issue with making the app "more holistic" is that is really code for pre-med style volunteer in a soup kitchen to show I'm Mother Theresa type bull****. No thanks. I didn't do it for med school apps and I'm definitely not doing that for residency apps. In fact, I would blackball anyone who is in student government or anything like that but that's my bias showing the other extreme view of that situation. If by holistic they mean spends time with family, does hobbies and isn't a weirdo then yeah we already have that and don't need pass/fail to observe that.
Click to expand...
Haha love some of the student government people literally nominating each other for these awards and straight up asking people “please give [X] a standing ovation” during an awards ceremony. It’s crigeworthy
 
  • Like
Reactions: 6 users
Members don't see this ad :)
Neopolymath said:
Another issue with making the app "more holistic" is that is really code for pre-med style volunteer in a soup kitchen to show I'm Mother Theresa type bull****. No thanks. I didn't do it for med school apps and I'm definitely not doing that for residency apps. In fact, I would blackball anyone who is in student government or anything like that but that's my bias showing the other extreme view of that situation. If by holistic they mean spends time with family, does hobbies and isn't a weirdo then yeah we already have that and don't need pass/fail to observe that.
Click to expand...
I think what Holistic means is understanding the challenges and adversities of everyone and every background, and recognizing how those adversities can contribute to ______ (fill in the blank)
 
chlorocruorin said:
At what point will residencies say we need to stop looking at step 1 scores to stratify applicants. Will my grandchildren need 280 to match into ortho and be deemed a "successful medical student".
Click to expand...

But competition over the most desirable residency spots will always exist (in fact, I think that it can only get worse over time), regardless of the form of step 1. If step 1 is pass/fail, what will our grandkids need to compete with the harvard med kids? A nature pub? An olympic medal? I think that it is more egalitarian to shoot for a high score on an academic exam than to gun for “holistic” things like that...
 
  • Like
Reactions: 6 users
I mean my main gripe is some people have better test taking skills than others. My knowledge base is fine, but sometimes I can’t figure out what the hell the question is asking. So it looks like I’m dumber. But I’m not looking much more than PC/ medium competitiveness so I don’t care too much

I think separating it into the P/HP/H, etc might be the way to go. That way there’s still scores to strive for, but it isn’t a big of a deal 220-230, but still a largedifference between the 200s and 250 people. Because a few points here and there is statistically insignificant anyway and it just adds to the stress of it all
 
  • Like
Reactions: 1 users
dohopeful13 said:
So what metrics are the alternative? Do start paying for specialty specific exams on top of everything else? Do we sit down and take 4 hour exams before each residency interview? Or do we accept that standardized testing is the norm and just grade step 1. Some honestly act like we haven't had to take a graded standardized test before and it's barbaric that it determines our future. The SAT, ACT, GRE, MCAT all exist. We've all taken them.
Click to expand...

Again, these are questions I cannot answer nor can any other member on here. Time will only tell. All we can do is hope (or complain) about a change none of us really have control over.
 
Chibucks15 said:
I think separating it into the P/HP/H, etc might be the way to go. That way there’s still scores to strive for, but it isn’t a big of a deal 220-230, but still a largedifference between the 200s and 250 people. Because a few points here and there is statistically insignificant anyway and it just adds to the stress of it all
Click to expand...
Yup... seems so easy. Could start that tomorrow.
 
  • Like
Reactions: 1 user
chlorocruorin said:
I think what Holistic means is understanding the challenges and adversities of everyone and every background, and recognizing how those adversities can contribute to ______ (fill in the blank)
Click to expand...
Being a competent note hoe intern year doesn't require a freaking hero coming of age story against adversity. It requires showing up on time (evaluated every step of the way), passing exams (specialty boards correlated with step 1), being polite (letters), and being normal (interviews.) Plenty of things in medical education to fix. This is not one of them.
 
  • Like
Reactions: 15 users
For residencies, one of the major goals is for all of your residents to pass their in-service exams. I imagine there is a strong correlation between step 1 score and ability to pass standardized boards. Hopefully the research discovers this and highlights exactly why step 1 is weighed so heavily
 
  • Like
Reactions: 1 user
  • Like
Reactions: 1 user
chlorocruorin said:
I think what Holistic means is understanding the challenges and adversities of everyone and every background, and recognizing how those adversities can contribute to ______ (fill in the blank)
Click to expand...

No. In GME land "holistic" just means turning off some of the screens and reading more of the apps.
 
  • Like
Reactions: 1 users
Members don't see this ad :)
Neopolymath said:
Being a competent note hoe intern year doesn't require a freaking hero coming of age story against adversity. It requires showing up on time (evaluated every step of the way), passing exams (specialty boards correlated with step 1), being polite (letters), and being normal (interviews.) Plenty of things in medical education to fix. This is not one of them.
Click to expand...

Well it looks like you got some work to do on the polite part if this is how you answer to those with an opposite view. Better start writing some more letters :)

"Another issue with making the app "more holistic" is that is really code for pre-med style volunteer in a soup kitchen to show I'm Mother Theresa type bull****. No thanks. I didn't do it for med school apps and I'm definitely not doing that for residency apps. In fact, I would blackball anyone who is in student government or anything like that but that's my bias showing the other extreme view of that situation. If by holistic they mean spends time with family, does hobbies and isn't a weirdo then yeah we already have that and don't need pass/fail to observe that."
 
Kracin said:
For residencies, one of the major goals is for all of your residents to pass their in-service exams. I imagine there is a strong correlation between step 1 score and ability to pass standardized boards. Hopefully the research discovers this and highlights exactly why step 1 is weighed so heavily
Click to expand...

too lazy to do the full lit search, but based on one retrospective cohort study that looked at factors which predicted eventual board certification, the score at which Step 1 predicts >93% likelihood of board certification is somewhere near 215

joc15083t1.jpeg

like the MCAT, the point at which you reach heavy diminishing returns on the usefulness of the exam for predicting future success is far below the point at which selection and differentiation is actually happening. In fact, Step 2 CK seems to be a marginally better predictor
 
  • Like
Reactions: 4 users
Med Ed said:
No. In GME land "holistic" just means turning off some of the screens and reading more of the apps.
Click to expand...
> We want a "holistic" system with less screens in place
> We want to reduce the amount of applications residencies receive
You may only pick 1
 
  • Like
  • Haha
Reactions: 3 users
Chibucks15 said:
I mean my main gripe is some people have better test taking skills than others. My knowledge base is fine, but sometimes I can’t figure out what the hell the question is asking. So it looks like I’m dumber. But I’m not looking much more than PC/ medium competitiveness so I don’t care too much

I think separating it into the P/HP/H, etc might be the way to go. That way there’s still scores to strive for, but it isn’t a big of a deal 220-230, but still a largedifference between the 200s and 250 people. Because a few points here and there is statistically insignificant anyway and it just adds to the stress of it all
Click to expand...

use the Duck Test :)

en.wikipedia.org

Duck test - Wikipedia

en.wikipedia.org en.wikipedia.org
 
chlorocruorin said:
Well it looks like you got some work to do on the polite part if this is how you answer to those with an opposite view. Better start writing some more letters :)

"Another issue with making the app "more holistic" is that is really code for pre-med style volunteer in a soup kitchen to show I'm Mother Theresa type bull****. No thanks. I didn't do it for med school apps and I'm definitely not doing that for residency apps. In fact, I would blackball anyone who is in student government or anything like that but that's my bias showing the other extreme view of that situation. If by holistic they mean spends time with family, does hobbies and isn't a weirdo then yeah we already have that and don't need pass/fail to observe that."
Click to expand...
Sorry, man. I'll hold the door for you next time we see each other in real life lol.
 
  • Like
  • Haha
Reactions: 4 users
chlorocruorin said:
Again, these are questions I cannot answer nor can any other member on here. Time will only tell. All we can do is hope (or complain) about a change none of us really have control over.
Click to expand...

This is the answer from every single person trying to propose a P/F Step 1. "I do not know what the answer is. I just know we should make Step 1 P/F and then wing it".
 
  • Like
  • Haha
Reactions: 9 users
Lucca said:
too lazy to do the full lit search, but based on one retrospective cohort study that looked at factors which predicted eventual board certification, the score at which Step 1 predicts >93% likelihood of board certification is somewhere near 215
Click to expand...

I get the feeling this is just bad analysis.
Comparing USMLE score and board cert is like comparing apples and oranges. The two aren't really related. If we did an experiment where we just threw people into specialties randomly then maybe it would tell us more. But we don't, we select based on score. So the 215 scorer doesn't end up in vascular surgery... they go to FM. But the 260 scorer can potentially do either and everyone ends up with a board cert, which further reinforces the error here.
 
  • Like
Reactions: 2 users
7331poas said:
This is the answer from every single person trying to propose a P/F Step 1. "I do not know what the answer is. I just know we should make Step 1 P/F and then wing it".
Click to expand...
I mean you don't even know the answers to your own questions lmao. Instead, you get upset and post about it on a forum. I'm actually just chilling, legally rolling up TBH lol
 
chlorocruorin said:
I mean you don't even know the answers to your own questions lmao. Instead, you get upset and post about it on a forum. I'm actually just chilling, legally rolling up TBH lol
Click to expand...
You must be new here
 
  • Like
Reactions: 1 user
chlorocruorin said:
I mean you don't even know the answers to your own questions lmao. Instead, you get upset and post about it on a forum. I'm actually just chilling, legally rolling up TBH lol
Click to expand...

The answer is literally don't do anything, keep percentiles so applicants can be easily stratified.
 
  • Like
Reactions: 6 users
TheRealCookieMonster said:
I get the feeling this is just bad analysis.
Comparing USMLE score and board cert is like comparing apples and oranges. The two aren't really related. If we did an experiment where we just threw people into specialties randomly then maybe it would tell us more. But we don't, we select based on score. So the 215 scorer doesn't end up in vascular surgery... they go to FM. But the 260 scorer can potentially do either and everyone ends up with a board cert, which further reinforces the error here.
Click to expand...

It's the data we have, obviously the conclusions one can draw are always limited by the data. It's not like you can sit a bunch of med students for Step 1 and then a week later ask them to take every single board cert. exam to learn how good of a predictor it is. There are many specialty specific studies such as this one, but one would have to look them up. The number you need to be competitive for a specialty changes with the times. Some specialties that are competitive today were not so 20-30 years ago when Step 1 scores were significantly lower than today and the exam was different. Were more physicians failing their boards? I dont have the data but my educated guess is that basically that rate has remained stable.

The actual analysis methods look fine to me, pretty typical for this kind of study. You can always write the JAMA editors to retract the article. Factors Associated With American Board of Medical Specialties Member Board Certification Among US Medical School Graduates
 
Last edited:
  • Like
Reactions: 1 user
This data is from the 1997-2000 cohort, which was ~20 years ago and Step has changed a lot since then. Cherry picking groupings in such a way prevents this data from being useful at all besides looking at boards pass/fail. This study doesn't even examine separating individuals by Step 1 score.
People shouldn't blindly follow the data but rather critically evaluate the study as they were taught to do.
 
  • Like
Reactions: 3 users
Slavic Scot said:
The answer is literally don't do anything, keep percentiles so applicants can be easily stratified.
Click to expand...
Its gotten to the point where they test on ridiculous **** in order to stratify though. Its moved past a medical competence exam. I'm tired of studying (and missing) questions that will never matter to me as a physician unless I go back and get my PhD. There's a better way. It ain't P/F, but some of the crap they test on now is insane. Not to mention deliberately wording questions and answers strangely, and in terms that none of us have ever heard, doesn't test our knowledge, it tests our ability to figure out what the hell the question wants.
 
  • Like
Reactions: 1 user
Sophist said:
This data is from the 1997-2000 cohort, which was ~20 years ago and Step has changed a lot since then. Cherry picking groupings in such a way prevents this data from being useful at all besides looking at boards pass/fail. This study doesn't even examine separating individuals by Step 1 score.
People shouldn't blindly follow the data but rather critically evaluate the study as they were taught to do.
Click to expand...

would it be nice to have a curve like we do for the MCAT with % graduating medical school at X score range? Yes. That said, this "cherry-picked' cohort is n=42,000, a fairly sizeable chunk of all 57,000 graduates in that time period. That merely a first-time passing score was sufficient to hit 93% board cert. rate or higher for the specialties listed should give you a pretty good hint of where the diminishing returns are on Step 1. Step 1 has changed but do we have any evidence that its effectiveness as a predictor of success has improved or worsened? One supposes that it has improved but in what way? Higher scores mean more and lower scores mean less? The rise in competition would have you believe so but I think it's a little wishful to actually believe that the state of affairs reproduced by the level of competition accurately reflects the utility of the exam; i.e., that a 260 is significantly more likely to pass ortho boards than a 220.

It's fine if you're against Step 1 P/F for a host of other reasons and complications associated with it, but please let's not pretend that the reason Step 1 P/F is bad is because we're concerned about physicians passing their specialty boards. I can almost guarantee nobody on this site cares about the %age of people passing their specialty boards. People are concerned that a changed exam will negatively impact their ability to compete for the most competitive residency positions/specialties.
 
Last edited:
  • Like
Reactions: 1 users
Lucca said:
You can always write the JAMA editors to retract the article.
Click to expand...
There's nothing wrong with the study. It just has little relevance here. Although, they did examine step score the primary concern of the study was really focused on pass/fail. The extra data about the mean of a board cert physician was even broken down by specialty. And naturally the peds physicians had lower scores than the IMs.
 
Lawper said:
Wait sorry, but are there any significant updates regarding this? Or we still don't know what's going to happen?
Click to expand...
forums.studentdoctor.net

Pass/ Fail Step 1

Had a path professor who diagnosed Goodpasture her second year of residency, nobody else knew what was going on, but some aspect of it triggered her memories it from step 1 studying. Even if a lot of that knowledge is consciously lost, some residue remains and can be helpful to patients with...
forums.studentdoctor.net forums.studentdoctor.net
New... But is it significant? hard to say lol
 
Lucca said:
would it be nice to have a curve like we do for the MCAT with % graduating medical school at X score range? Yes. That said, this "cherry-picked' cohort is n=42,000, a fairly sizeable chunk of all 57,000 graduates in that time period. That merely a first-time passing score was sufficient to hit 93% board cert. rate or higher should give you a pretty good hint of where the diminishing returns are on Step 1.
Click to expand...
Disagree. They're bucketing Step 1 pass together, which does not tell whether there is increased board certification rate with higher score than pass score. Why 93%? That's such a strange number. Why didn't they attempt to find the point at which there is 95%, or some arbitrary % certification?

Lucca said:
It's fine if you're against Step 1 P/F for a host of other reasons and complications associated with it, but please let's not pretend that the reason Step 1 P/F is bad is because we're concerned about physicians passing their specialty boards.
Click to expand...
Lucca said:
like the MCAT, the point at which you reach heavy diminishing returns on the usefulness of the exam for predicting future success is far below the point at which selection and differentiation is actually happening. In fact, Step 2 CK seems to be a marginally better predictor
Click to expand...

Passing specialty boards is not even a main issue that we brought up against Pass/Fail Step 1. The main issues are that there will be increasing emphasis on nepotism, medical school prestige, and other subjective factors. No one is "pretending that the reason Step 1 P/F is bad is because we're concerned about physicians passing their specialty boards".
 
  • Like
Reactions: 3 users
aProgDirector said:
I too reviewed the report. I expected a decision of some sort. They made no decision, just wrote down a list of reasonable options, and then said that other groups should get together and make some sort of a decision. And that we should all just agree and be happy.

So, basically, nothing. They say we should convene a UME/GME panel and make a decision by 2019.

Seems like they convened a panel who decided that there needs to be a panel to make a decision.
Click to expand...

Thanks for the summary. The arguments and discussions were getting repetitive so wasn't sure.
 
Sophist said:
Disagree. They're bucketing Step 1 pass together, which does not tell whether there is increased board certification rate with higher score than pass score. Why 93%? That's such a strange number. Why didn't they attempt to find the point at which there is 95%, or some arbitrary % certification?




Passing specialty boards is not even a main issue that we brought up against Pass/Fail Step 1. The main issues are that there will be increasing emphasis on nepotism, medical school prestige, and other subjective factors. No one is "pretending that the reason Step 1 P/F is bad is because we're concerned about physicians passing their specialty boards".
Click to expand...

my original post was in response to someone commenting that step 1 correlates with board cert performance
 
  • Like
Reactions: 1 user
If Step was p/f, I’d be hitting 3rd year not knowing anything. There’s a lot of crap that shouldn’t be on it. I.e. I’m probably going to do well in biochem. I have no idea why I should have an advantage over applicants that know things that matter, but whatever.

Literally no one comes into med school thinking they won’t have to perform well on boards. If they chose to focus on their well being or go on vacation in the middle of a block or whatever else while I sat in a study room smashing the space bar for 4-6 hrs/day, that’s fine with me. In fact, plenty of those folks are gonna outperform me. That’s okay. They knew what they needed to do and did it. But I honestly have no clue why anyone thinks that people who spent less time studying than they should have, and then prove on a test that they know less information as a result, should have the same opportunities as people who worked harder.

It’d be great if we just averaged 1 & 2 together and reported that though.
 
  • Like
Reactions: 7 users
aProgDirector said:
I too reviewed the report. I expected a decision of some sort. They made no decision, just wrote down a list of reasonable options, and then said that other groups should get together and make some sort of a decision. And that we should all just agree and be happy.

So, basically, nothing. They say we should convene a UME/GME panel and make a decision by 2019.

Seems like they convened a panel who decided that there needs to be a panel to make a decision.
Click to expand...

also quick question: do these organizations plan on getting rid of Step 2 CS? because that exam is pass/fail and is hugely despised on here for being a complete waste of time and money, on top of being unduly stressful.
 
  • Like
Reactions: 1 user
what a useless position paper. Limiting # of applications will only hurt weak students who need to apply to lots of programs to match. If I had a limited # of apps, I wouldn't even have applied to the program I matched at because I'd consider myself too low tier to match there. Strong students won't be impacted by limiting apps to 50, but any student with a pre clinical class failure, low board score, LOA will be hurt by limiting themselves to 50 apps in something like low to mid tier IM. I don't understand why it's NBMEs duty to police how many programs students apply.
 
  • Like
Reactions: 3 users
Lucca said:


I think this is a nice summary of the position counter to the echo chamber on here/reddit.
Click to expand...


Why do you assume it's an echo chamber? You really don't think it might be possible that many students oppose these proposed changes?
 
  • Like
Reactions: 1 users
ricksimmonsrafowens said:
Why do you assume it's an echo chamber? You really don't think it might be possible that many students oppose these proposed changes?
Click to expand...

I think *most* students oppose these changes based on the surveys that have been published previously on this general question. But I think the opposing viewpoints, which drove the precipitation of this meeting, the recommendations etc., have been under and misrepresented over and over again circularly.
 
  • Like
Reactions: 1 user
Lucca said:
I think *most* students oppose these changes based on the surveys that have been published previously on this general question. But I think the opposing viewpoints, which drove the precipitation of this meeting, the recommendations etc., have been under and misrepresented over and over again circularly.
Click to expand...
This isn't journalism. We don't have to utilize a misguided fairness doctrine.
 
  • Like
Reactions: 1 user
Status
Not open for further replies.
Next unread thread

Similar threads

V
Antecdotally, does it seem like school tier is playing a more important role now that step 1 is pass fail?
2
Replies
61
Views
6K
goldenfloofers
G
Lifeblood_20
MD Failing Step 1
Replies
27
Views
7K
Lifeblood_20
Lifeblood_20
A
Pros vs. cons of pass/fail clinical years
Replies
34
Views
3K
i-doctor
I
D
I fail to understand why step matters so much....
2 3 4
Replies
193
Views
11K
wingingit
W
C
Should I take step 1 scored or pass/fail?
Replies
22
Views
1K
Colinoscopy
C
Top