Pass/ Fail Step 1

[Neopolymath]

---

Members don't see this ad.

it's over an hour...

also interesting to note that the proponents of making Step 1 pass/fail are also thinking of converting Step 2 CS from pass/fail to a scored exam.

This is really all you need to know about the type of person who thinks this is a good idea.

 

[Lawpy]

This is really all you need to know about the type of person who thinks this is a good idea.

I read in a related discussion that in the end, these organizations actually don't care what students think so they'll just change whatever they see fit. Meanwhile, the students will continue to suffer as they try to navigate through this new system.

And people continue to wonder why students get jaded...

 

[Med Ed]

it's over an hour...

OMG.

also interesting to note that the proponents of making Step 1 pass/fail are also thinking of converting Step 2 CS from pass/fail to a scored exam.

Where did you see that?

 

[Med Ed]

My residency application experience in a nutshell:

Applied to 20 programs
Received 19 invites
Went to 12
Ranked 12
Matched #2

First-come-first-serve mass interview invites weren't invented. Overall match rate was the same as it is now (~94%). How anyone could think the present situation is better is beyond me.

 

[Lawpy]

OMG.



Where did you see that?

Would be better to have a detailed article outlining the counterargument rather than having to watch a long video but that's just me.

Regarding Step 2 CS:

Pass/ Fail Step 1

Another issue with making the app "more holistic" is that is really code for pre-med style volunteer in a soup kitchen to show I'm Mother Theresa type bull****. No thanks. I didn't do it for med school apps and I'm definitely not doing that for residency apps. In fact, I would blackball anyone...

forums.studentdoctor.net

Was highlighted in this post

 

[ricksimmonsrafowens]

My residency application experience in a nutshell:

Applied to 20 programs
Received 19 invites
Went to 12
Ranked 12
Matched #2

First-come-first-serve mass interview invites weren't invented. Overall match rate was the same as it is now (~94%). How anyone could think the present situation is better is beyond me.

Presently, if I apply to more programs I have more choices of where to apply, opportunities to look at programs websites and see if it's the right program for me... There's nothing wrong with applying to more or less than 20 programs. What field did you apply to btw? If it's IM you can still match with 20 apps today. Idk why # of programs applied to is such a big issue. Just like in the good old days PDs still filter based on steps, med school, location, LORs, etc

 

[Med Ed]

Would be better to have a detailed article outlining the counterargument rather than having to watch a long video but that's just me.

Regarding Step 2 CS:

Pass/ Fail Step 1

Another issue with making the app "more holistic" is that is really code for pre-med style volunteer in a soup kitchen to show I'm Mother Theresa type bull****. No thanks. I didn't do it for med school apps and I'm definitely not doing that for residency apps. In fact, I would blackball anyone...

forums.studentdoctor.net

Was highlighted in this post

If you actually read the document you will note that the bullet points were compiled as topics of conversation amongst conference attendees. They do not represent the opinions or positions of the NBME itself (which in fact is very much in favor of maintaining the status quo).

 

[Med Ed]

Idk why # of programs applied to is such a big issue.

Ask the program directors. They're the ones who are getting crushed by applications. Which makes them turn to metrics to whittle down the applicant pools to reasonable levels. Which places increased emphasis on Step 1 scores. Which leads to the present "Step 1 climate." Which leads to InCUS being called. Which leads to numerous threads on Reddit and SDN about the possibility of a P/F Step 1.

 

[Lawpy]

If you actually read the document you will note that the bullet points were compiled as topics of conversation amongst conference attendees. They do not represent the opinions or positions of the NBME itself (which in fact is very much in favor of maintaining the status quo).

I mean from what I gathered, no decision was made currently. However, the fact that these suggestions were even outlined in the first place are concerning, as they didn't make much sense. I used Step 2 CS as an example of a pass/fail Step that still has a lot of problems, meaning that pass/fail Step 1 won't be necessarily better.

Do you know roughly when we'll get the final decision?

 

[Med Ed]

I mean from what I gathered, no decision was made currently. However, the fact that these suggestions were even outlined in the first place are concerning.

Why would something discussed at a "village fair" station at InCUS be concerning? The whole purpose of the exercise was to give attendees the illusion of input.

I used Step 2 CS as an example of a pass/fail Step that still has a lot of problems, meaning that pass/fail Step 1 won't be necessarily better.

Comparing any of the MCQ Steps to CS is apples to Buicks. As has been pointed out ad nauseum, if Step 1 went P/F then emphasis would shift to Step 2 CK, which (IMHO) is likely a better measure for residency selection.

Do you know roughly when we'll get the final decision?

Well, you don't want to listen and you don't want to read. How about a click?

 

[Med Ed]

Would be better to have a detailed article outlining the counterargument rather than having to watch a long video but that's just me.

The Sheriff of Sodium

Salty about medical education

thesheriffofsodium.com

 

[Lawpy]

Why would something discussed at a "village fair" station at InCUS be concerning? The whole purpose of the exercise was to give attendees the illusion of input.

It just seems odd they even considered changing a pass/fail exam to a scored exam when the leading focus is to make an existing scored exam pass/fail. Even if it's just meant for everyone to give their input (or the illusion of it), the fact that they even thought that making Step 2 CS a scored exam is a reasonable suggestion is strange.

Comparing any of the MCQ Steps to CS is apples to Buicks. As has been pointed out ad nauseum, if Step 1 went P/F then emphasis would shift to Step 2 CK, which (IMHO) is likely a better measure for residency selection.

It was just meant to show that pass/fail exams aren't necessarily better. Also what would happen to Step 2 CS if Step 1 went P/F?

Well, you don't want to listen and you don't want to read. How about a click?

Thanks for the link 😛 But like I mentioned earlier, nothing much of consequence happened yet but hopefully we get more concrete insights later this year. Maybe nothing will happen and things will continue the way they currently are.

I agree the system as it is has a lot of problems and Step 1 Mania is a real issue. I'm just not sure if converting it to P/F will help.

The Sheriff of Sodium

Salty about medical education

thesheriffofsodium.com

Thanks

 

[Med Ed]

It just seems odd they even considered changing a pass/fail exam to a scored exam when the leading focus is to make an existing scored exam pass/fail. Even if it's just meant for everyone to give their input (or the illusion of it), the fact that they even thought that making Step 2 CS a scored exam is a reasonable suggestion is strange.

Who do you think “they” are?

 

[Stagg737]

You really think neurosurgery is going to put out a product of similar quality that is constantly being updated with different questions throughout the year. Multiply this by every single specialty. It's unsustainable, especially when we are already doing shelf exams that fill a similar role.

Sorry, but this is one of the dumbest things I've read in a while. What do you think specialty board exams are? You could literally just have applicants to a specialty take the board exam for their field and grade on a curve accordingly.

This is really all you need to know about the type of person who thinks this is a good idea.

If they could come up with a consistent and comprehensive grading system which would be scored by people who actually understood what is relevant to practice and not just actors, then I'd argue that Step 2CS would be far more valuable than any of the other steps. CS/PE are infinitely more relevant and consistent with what we do on a daily basis as residents than the written exams and I can genuinely say that less than 10% of what was tested on Step/Level 1 and 2 is relevant to what I do on a daily basis. They're fine tests if you're going to be a generalist (peds, FM, IM), but if you're entering a specialty or sub-specialty the amount of extraneous information in those exams is ridiculous.

The problem is the ridiculous inconsistency and garbage grading scale of CS. I really wish they would standardize it more, but the cost, both financially and in terms of manpower, of doing this is not realistic. So that will just stay in my fantasy world where logical things actually happen for the rest of forever.

 

[Dr. Doctor MD]

Would be better to have a detailed article outlining the counterargument rather than having to watch a long video but that's just me.

Regarding Step 2 CS:

Pass/ Fail Step 1

Another issue with making the app "more holistic" is that is really code for pre-med style volunteer in a soup kitchen to show I'm Mother Theresa type bull****. No thanks. I didn't do it for med school apps and I'm definitely not doing that for residency apps. In fact, I would blackball anyone...

forums.studentdoctor.net

Was highlighted in this post

I've finished his video and have read several articles on his blog, and he makes some astronomical leaps to draw the conclusions that he states (mostly on his blog). A lot of it is pure conjecture or theory with loose associations from references to support his argument. You can't argue with someone who picks a viewpoint and then searches out "data" to try to support it. If you read his blogs, you can quickly see that a lot of the assumptions and links he makes are unfounded and may sound good to the uneducated reader, but simply don't make sense in reality.

EDIT: To his credit, I will say that after reading many more of his posts, he oscillates rapidly between making really good points and arguments at times and then moving onto looser unfounded arguments.

 

[mrbreakfast]

The problem is the ridiculous inconsistency and garbage grading scale of CS.

The extension of this is ultimately the central argument for standardized testing (Step 1/CK/shelves): the fact that they're standardized.

Clinical grades and CS are incredibly relevant to clinical practice, in theory. In reality, it's a mess of grade inflation, attendings who evidently fill out 30 evaluations in 5 minutes, and residents who effectively grade you on your personability.

If we earnestly want to improve the residency application process, we need to first address the system where simply being physically present can earn you a mix of "above average" and "outstanding" marks and where the comment "student worked hard and performed at the level expected in every category" is considered a negative evaluation.


I would be curious to know how PDs interpret clinical grades and the MSPE comments given the florid praise that seems to be commonplace.

 

[FindersFee5]

My residency application experience in a nutshell:

Applied to 20 programs
Received 19 invites
Went to 12
Ranked 12
Matched #2

First-come-first-serve mass interview invites weren't invented. Overall match rate was the same as it is now (~94%). How anyone could think the present situation is better is beyond me.

And making step 1 pass/fail wouldn't fix this.

This is my personal crusade right now, but I think there is a really simple solution: let programs see the number of places an applicant has applied to, although not the specific programs. This rewards applicants who apply more selectively because programs know you're more worth an interview spot if you applied to 10-15 programs rather than 80. It makes residency selection easier because you have a smaller pool of people applying. It makes applicants need to choose their spots and think about their list more carefully. Finally, it actually provides some competitive incentive to apply to less programs (vs just time/money which when calculated against not matching, is a trivial investment).

 

[Med Ed]

And making step 1 pass/fail wouldn't fix this.

Agree, just thought a little personal anecdote would help underscore the system's present level of dysfunction. Residency over-application and Step 1 mania are separate issues, although they are interrelated.

This is my personal crusade right now, but I think there is a really simple solution: let programs see the number of places an applicant has applied to, although not the specific programs. This rewards applicants who apply more selectively because programs know you're more worth an interview spot if you applied to 10-15 programs rather than 80. It makes residency selection easier because you have a smaller pool of people applying. It makes applicants need to choose their spots and think about their list more carefully. Finally, it actually provides some competitive incentive to apply to less programs (vs just time/money which when calculated against not matching, is a trivial investment).

I have found this to be an intriguing idea, but IMHO the implementation would be chaotic because of information asymmetry. This approach would given applicants competing demands (more opportunities vs. better odds per opportunity) with no basis to choose a particular number of applications other than guessing (after spending many hours on Reddit, of course). After a couple of years I imagine some specialty-specific heuristics might become apparent.

My preference is a tiered system, allowing applicants to apply to a limited number of programs initially (the exact number would be defined by each specialty) and then later opening it up for unlimited applications. That would give the PD's useful information while preserving some degree of choice for applicants. Not perfect, but perhaps better than what we have now.

Whatever happens, I don't think any change is really workable until Residency Exploration Tool comes online and proves itself to be useful.

 

[Slavic Scot]

The Sheriff of Sodium

Salty about medical education

thesheriffofsodium.com

"Maybe your Step 1 score opened a door for you. But the the key ingredient wasn’t Step 1 – it was you.

Look, you did what you had to do. We told you to memorize basic science, and memorize it you did. We told you that Step 1 scores were the most important thing in your application, so you hit Step 1 with everything you had. You succeeded in the gauntlet that we put you through, and you should rightly be proud of the work you put in and the knowledge that you gained in doing so.

Making Step 1 pass/fail doesn’t take away your accomplishment. But it does make it possible to create a system in which future versions of you can distinguish themselves in challenges that will leave them better prepared for their future."

Thanks

Making Step 1 pass/fail would quite effectively take away that accomplishment, and replace it with nepotism, school prestige, and other subjective criteria. I haven't seen an argument that says otherwise.

Also, the more I think about it, the more I object to the term "accomplishment". Standardized tests are not accomplishments, they are tools to see where you stand in relation to other students.

 

[Med Ed]

Making Step 1 pass/fail would quite effectively take away that accomplishment, and replace it with Step 2 CK.

FTFY.

 

[deleted938090]

Let's revert preclinical grades from pass fail now we're at it. Seems like that would make students listen to their courses. The more we put under pass/fail, the more students will emphasize the aspects that are not under pass/fail. Who knows; if this passes, people would have to focus on away rotations or on publication number to compete.
Making Step 1 P/F won't change the kinds of things mentioned in the video.
"____ is the difference between having the option to do what I want to do for the rest of my life and a compromise. Anything that gets in the way of that doesn't seem helpful at the moment."
Replace the blank in this video with anything else after Step 1 and the problem still stands.

And if people think multiple choice isn't a good way of assessment, they should increase the number of choices and include a "None of the above" choice for every question.
As the original commentator of the USMLE has said:

The questions presented as examples of how "esoteric" USMLE questions have become aren't very convincing either (chemotactic factor and the recombination in lymphocytes are simple reasoning); besides the desmopressin question (which sounds a lot like vasopressin which led me to guess adenylyl cyclase), they are well within the undergrad curriculum and I'm fairly certain medical school will cover it.
The arguments of opportunity costs are not valid, either, because before people like Collins or Margaret Hamilton could do their work, they had to learn things too. It is insincere to say that students do not need to learn the content before they could do something that benefits society.

 

[Lucca]

Whose thinking is more magical, mine that says in the face of a substantial change new and more thoughtfully engineered metrics will emerge to evaluate residency candidates that will improve on the current model, as has already been done in Emergency Medicine to some success apparently, or those that think that PDs are going to keep USNWR open in a separate tab so they can decide which apps to throw in the shredder that year without reading them.

 

[deleted938090]

Whose thinking is more magical, mine that says in the face of a substantial change new and more thoughtfully engineered metrics will emerge to evaluate residency candidates that will improve on the current model, as has already been done in Emergency Medicine to some success apparently, or those that think that PDs are going to keep USNWR open in a separate tab so they can decide which apps to throw in the shredder that year without reading them.

This will continue to happen as long as there is competition for certain coveted positions. If they change to "more thoughtfully engineered metrics", then those who do poorly on those will have their applications tossed, too.

 

[Goro]

I read in a related discussion that in the end, these organizations actually don't care what students think so they'll just change whatever they see fit. Meanwhile, the students will continue to suffer as they try to navigate through this new system.

And people continue to wonder why students get jaded...

They sure as hell don't care about your career aspirations

 

[Slavic Scot]

Whose thinking is more magical, mine that says in the face of a substantial change new and more thoughtfully engineered metrics will emerge to evaluate residency candidates that will improve on the current model, as has already been done in Emergency Medicine to some success apparently, or those that think that PDs are going to keep USNWR open in a separate tab so they can decide which apps to throw in the shredder that year without reading them.

Does it though? I don't know how you define success here, frankly. If you look at charting outcomes, you'd never be able to tell there was a separate standardized (?) exam (?) for matching EM.

 

[5yewy5r]

Whose thinking is more magical, mine that says in the face of a substantial change new and more thoughtfully engineered metrics will emerge to evaluate residency candidates that will improve on the current model, as has already been done in Emergency Medicine to some success apparently, or those that think that PDs are going to keep USNWR open in a separate tab so they can decide which apps to throw in the shredder that year without reading them.

I don’t think the benefit of school name comes from referencing USNWR. Seems more likely that it’d be an amalgamation of a halo effect and having well-known letter writers with flashy and exciting research projects. In addition to this is the fact that stratification runs dry at some point, so inevitably school name plays a role.

TBQH, both of your lines of thought seem a bit magical, or perhaps naive to me? I mean this in no offense, I just want to have a discussion. What are the potentially better engineered metrics? Step 2? Most seem to consider that exam with contempt. And pushing forward with making step 1 P/F without concrete alternatives in mind seems foolhardy, a la Brexit. I don’t see how residency selection can avoid becoming like undergrad/med school selection once you remove the objective stratifier.

The EM SLOE system sounds promising if it can actually maintain consistency and objectivity.

 

[Lucca]

This will continue to happen as long as there is competition for certain coveted positions. If they change to "more thoughtfully engineered metrics", then those who do poorly on those will have their applications tossed, too.

I don’t doubt apps will always be tossed. The question is: how good of a metric is Step 1 to achieve that? The argument from many is that it’s better than anything else and some even say it is, in fact, very good. I don’t find those arguments very convincing, of course with the caveat that I must admit my own ignorance not having experienced the exam. But we’re talking about an exam with a 95% CI of +\- 16 which focuses on basic sciences. That’s the best one size fits all metric we can come up with to judge candidates for every single specialty? Seems like a rly lazy attitude to me.

 

[Lucca]

I don’t think the benefit of school name comes from referencing USNWR. Seems more likely that it’d be an amalgamation of a halo effect and having well-known letter writers with flashy and exciting research projects. In addition to this is the fact that stratification runs dry at some point, so inevitably school name plays a role.

TBQH, both of your lines of thought seem a bit magical, or perhaps naive to me? I mean this in no offense, I just want to have a discussion. What are the potentially better engineered metrics? Step 2? Most seem to consider that exam with contempt. And pushing forward with making step 1 P/F without concrete alternatives in mind seems foolhardy, a la Brexit. I don’t see how residency selection can avoid becoming like undergrad/med school selection once you remove the objective stratifier.

The EM SLOE system sounds promising if it can actually maintain consistency and objectivity.

No offense taken, I have no horse in this race I just, in general, am against high stakes standardized exams because I don’t think they very meaningfully measure anything *beyond a certain point* which is always below the range where it’s being used except for predicting performance on future standardized exams.

It’s not as if I’m involved with anything in Med Ed or GME or have any experience. I just have an opinion. But I also am not so cynical as to think GME/PDs will sit by and do nothing if the Step 1 filter is taken away, but I am cynical enough to believe they will do nothing until it is.

 

[7331poas]

I don’t doubt apps will always be tossed. The question is: how good of a metric is Step 1 to achieve that? The argument from many is that it’s better than anything else and some even say it is, in fact, very good. I don’t find those arguments very convincing, of course with the caveat that I must admit my own ignorance not having experienced the exam. But we’re talking about an exam with a 95% CI of +\- 16 which focuses on basic sciences. That’s the best one size fits all metric we can come up with to judge candidates for every single specialty? Seems like a rly lazy attitude to me.

Step 1 does not "focus on basic sciences". In fact the only basic science on the exam is a section for biochemistry.

 

[Med Ed]

Step 1 does not "focus on basic sciences".

Interesting take on things.

 

[Goro]

Step 1 does not "focus on basic sciences". In fact the only basic science on the exam is a section for biochemistry.

poas, just to verify, you have taken Step I, correct?

If so, what did Step I focus on, if not the basic sciences

 

[7331poas]

poas, just to verify, you have taken Step I, correct?

If so, what did Step I focus on, if not the basic sciences

cardiology microbiology renal physiology. etc.

I consider biology chemistry and physics to be basic sciences.

And no, I am scheduled to start dedicated soon however.

 

[Chibucks15]

cardiology microbiology renal physiology. etc.

I consider biology chemistry and physics to be basic sciences.

And no, I am scheduled to start dedicated soon however.

You’re passionate for someone who hasn’t actually experienced anything. Come back to us in a bit before you tell us what’s on it. I did the same thing as you and my eyes got opened wide

 

[7331poas]

You’re passionate for someone who hasn’t actually experienced anything. Come back to us in a bit before you tell us what’s on it. I did the same thing as you and my eyes got opened wide

If its drastically different than Uworld or the NBMEs then you may be correct. But even if it is, that does not change the spirit of my argument. The test could ask me to run my fastest mile time. You better believe I would start training if that was my only way to differentiate myself from other applicants.

 

[Goro]

cardiology microbiology renal physiology. etc.

I consider biology chemistry and physics to be basic sciences.

And no, I am scheduled to start dedicated soon however.

Okay, I think we are talking past each other in our definitions.

To me the basic Sciences are the course work that you take in your preclinical courses.

You are defining them as the mere prerequisites for doing well on MCAT. Mini Med schools are moving away from prerequisites to recommendations for competency.

It's a no-brainer that step one is not going to ask you about grignard reactions, or vectorial analysis

 

[Sephirakra]

Okay, I think we are talking past each other in our definitions.

To me the basic Sciences are the course work that you take in your preclinical courses.

You are defining them as the mere prerequisites for doing well on MCAT. Mini Med schools are moving away from prerequisites to recommendations for competency.

It's a no-brainer that step one is not going to ask you about grignard reactions, or vectorial analysis

What's the tallest applicant that Mini Med schools will take?

 

[AnatomyGrey12]

As someone who has now taken Step 1 my opinion has not changed. It is very clinically relevant, as most of my questions pertained to pharm, pathophys, path, phys, understanding basic biostats, very clinically relevant anatomy, basic next step management, etc. I still have yet to see an argument for a change in the current model that holds any water, let alone a good reason for this test to go P/F.

Change for the sake of change instead of the right change, will have devastating unforeseen consequences for large swaths of medical students.

 

[JooceMan137]

Im personally against the PF. I’m curious if making it PF would indeed put students from less prestigious schools at a disadvantage?

It seems like that would be the case, but most of the pro-PF propaganda says that this just a myth.

I’m interested to hear what people have to say about this.

 

[AnatomyGrey12]

most of the pro-PF propaganda says that this just a myth.

Most pro-PF people come from top schools so they have no idea what they are talking about with regards to this. I have yet to see on op-ed piece on this topic that wasn't written by someone at a T20.

 

[olabrador23]

Most pro-PF people come from top schools so they have no idea what they are talking about with regards to this. I have yet to see on op-ed piece on this topic that wasn't written by someone at a T20.

Common sense tells me that it is not that they do not know, they are only defending that position b/c they can gain from it.

 

[NotAProgDirector]

Went away for the weekend. Lots to get caught up on.

How would pass/fail step 1 change how you view students from lower-prestige schools? Do percentiles help those students get their foot in the door?

If P/F for S1 were to come to pass, programs would need to make decisions using other factors. S2 might be used, forcing students to take it earlier. I don't think we're ever going back to a world where there is no standardized exam to take. The big fields will make their own, if that's what's needed.

When would be the soonest they could implement whatever changes they decide on, January?

There is no clear timeline. The "plan" is to convene some sort of a group to review all of these issues, and (theoretically) make a decision in 2019-2020. But getting consensus is going to be difficult, if not impossible. Even if a decision is made in this timeframe, any change would be delayed for a few years. This isn't worth worrying about.

Why can’t we also assume that residency programs will look more holistically at applicants as a result of P/F instead of assuming the worse.

Answered by some others already, but depends on what you mean by "holistically". MSPE's do stratify students into groups. LOR's are not very helpful -- all say the student is in the top 5%, and relying on LOR's becomes a "who you know" / "How connected you are" game. So all we're left with are clinical grades and MSPE.

And, that's for US grads. 50% of IM spots are filled by IMG's, and many of them have useless MSPE's. All they have are their USMLE scores.

I think separating it into the P/HP/H, etc might be the way to go.

One of the options was to divide into quartiles or quintiles. It would be fine as far as I'm concerned. Problem is, if you're in that bottom group, I can't tell if you just barely passed, or if your performance was better than that. Some programs will assume the worst.

No. In GME land "holistic" just means turning off some of the screens and reading more of the apps.

We've had this discussion before, but 1) that might not be possible, and 2) this just elevates the importance of other parts of the application which may be just as flawed, or worse. Plus, IMG's fill a large number of spots in the US, and their applications may be mostly worthless except for exam scores. And if we just switch to S2 scores, we're just pushing the problem onto a new exam (and students will want "dedicated" 8 weeks to study for it).

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3217584/pdf/nihms331403.pdf

This data is from the 1997-2000 cohort, which was ~20 years ago and Step has changed a lot since then. Cherry picking groupings in such a way prevents this data from being useful at all besides looking at boards pass/fail. This study doesn't even examine separating individuals by Step 1 score.
People shouldn't blindly follow the data but rather critically evaluate the study as they were taught to do.

This study often comes up in this discussion. It was done looking at graduates in 1997-2000, when step scores were lower (av S1 was 215). The USMLE "states" that scores are equivalent over time. Either that's wrong and there's some inflation over time, or we licensed a bunch of people in the past who should have failed (as the minimum pass on S1 has risen from 175 to 194 -- you can see the trends here: A peek inside the USMLE sausage factory: setting the Step 1 minimum passing score). I think it's the former.

So, problems with this study:
1. If 215 => 93% board pass, that's a 215 in 1997 which was the average. It certainly isn't a 215 today.
2. A 93% board pass rate would put an IM program in the 3rd or 4th quartile (bottom) of IM programs. That is a terrible board pass rate.

So it's an interesting study, but limited.

It’d be great if we just averaged 1 & 2 together and reported that though.

This is one of the options listed. Would students be required to take both? Or could they just take S1 and if "happy", just plan to take S2 later (much like today)? And since S2 scores are higher than S1 scores on average, this will just lead to further score inflation.

It just seems odd they even considered changing a pass/fail exam to a scored exam when the leading focus is to make an existing scored exam pass/fail.

It's a tradeoff, and they are trying to have it both ways. Since CS is P/F, if you fail you can't show how badly you failed (just barely vs miserably), and then when you subsequently pass can't show whether you passed barely or by a mile. Adding scoring would help that, but then we're back to S1 mania issues with scores.

I would be curious to know how PDs interpret clinical grades and the MSPE comments given the florid praise that seems to be commonplace.

It's an enormous problem. Everyone is in the top 5% in LOR's. Many MSPE's are at least more reasonable, but some continue to place students into "groups" but then don't tell you how many are in each group.

let programs see the number of places an applicant has applied to, although not the specific programs.

I have found this to be an intriguing idea, but IMHO the implementation would be chaotic because of information asymmetry.

My preference is a tiered system, allowing applicants to apply to a limited number of programs initially (the exact number would be defined by each specialty) and then later opening it up for unlimited applications. That would give the PD's useful information while preserving some degree of choice for applicants. Not perfect, but perhaps better than what we have now.

Whatever happens, I don't think any change is really workable until Residency Exploration Tool comes online and proves itself to be useful.

Either is a reasonable option. I agree the first will drive students crazy trying to determine what number of applications is "best". And the latter, some fields may decide to only review the preference apps, placing a defacto hard limit on applications and forcing students to decide which programs they are really competitive for.

The explorer tool may end up being useful, probably better than where we are now.

The EM SLOE system sounds promising if it can actually maintain consistency and objectivity.

Sure, but then your entire career hangs in the balance of a single evaluation. In the ED you work with a mix of people so perhaps it's a bit more fair. And schools would need to ensure that not 95% of student get the top rating. Soon students will be complaining that the evaluations are subjective and unfair, and that they want an objective system. Plus this create a situation where you only get your eval when your training is just about done -- if it's poor, you have little time to reboot your career.

And, not useful for IMG's. So really not a system wide answer.

But we’re talking about an exam with a 95% CI of +\- 16 which focuses on basic sciences.

Might be taking your comment out of context (sorry!), but this is also a common statement -- that because the SD is wide enough that most scores can't be compared. This is not exactly accurate. First, these statistics are used to compare the averages of two groups to see if they come from the same population, not comparing two individual scores. Second, this all assumes a 95% confidence interval -- I'm happy with a much lower level of confidence. The number quoted (16?) - not sure where that is from. If it's the standard dev of the S1 score distribution, that's not what you want.

My bottom line:
1. If the FSMB and the AAMC want to change to P/F, there's no one to stop them.
2. As they seem to have punted this to some still-to-be-determined group of concerned citizens, I doubt they will be able to come to any consensus.
3. As a non trusting sort, I would move ahead with development of a specialty specific exam so that, as a field, we simply don't need to worry about this any more. Likely outcome is --> group formed --> lots of talk --> no agreement --> AAMC/NBME/FSMB decide whatever they want anyway.

 

[Slavic Scot]

My bottom line:
1. If the FSMB and the AAMC want to change to P/F, there's no one to stop them.
2. As they seem to have punted this to some still-to-be-determined group of concerned citizens, I doubt they will be able to come to any consensus.
3. As a non trusting sort, I would move ahead with development of a specialty specific exam so that, as a field, we simply don't need to worry about this any more. Likely outcome is --> group formed --> lots of talk --> no agreement --> AAMC/NBME/FSMB decide whatever they want anyway.

Thanks! It feels good there's someone on your side/preparing for the future.

 

[olabrador23]

Went away for the weekend. Lots to get caught up on.



If P/F for S1 were to come to pass, programs would need to make decisions using other factors. S2 might be used, forcing students to take it earlier. I don't think we're ever going back to a world where there is no standardized exam to take. The big fields will make their own, if that's what's needed.



There is no clear timeline. The "plan" is to convene some sort of a group to review all of these issues, and (theoretically) make a decision in 2019-2020. But getting consensus is going to be difficult, if not impossible. Even if a decision is made in this timeframe, any change would be delayed for a few years. This isn't worth worrying about.



Answered by some others already, but depends on what you mean by "holistically". MSPE's do stratify students into groups. LOR's are not very helpful -- all say the student is in the top 5%, and relying on LOR's becomes a "who you know" / "How connected you are" game. So all we're left with are clinical grades and MSPE.

And, that's for US grads. 50% of IM spots are filled by IMG's, and many of them have useless MSPE's. All they have are their USMLE scores.


One of the options was to divide into quartiles or quintiles. It would be fine as far as I'm concerned. Problem is, if you're in that bottom group, I can't tell if you just barely passed, or if your performance was better than that. Some programs will assume the worst.

We've had this discussion before, but 1) that might not be possible, and 2) this just elevates the importance of other parts of the application which may be just as flawed, or worse. Plus, IMG's fill a large number of spots in the US, and their applications may be mostly worthless except for exam scores. And if we just switch to S2 scores, we're just pushing the problem onto a new exam (and students will want "dedicated" 8 weeks to study for it).

This study often comes up in this discussion. It was done looking at graduates in 1997-2000, when step scores were lower (av S1 was 215). The USMLE "states" that scores are equivalent over time. Either that's wrong and there's some inflation over time, or we licensed a bunch of people in the past who should have failed (as the minimum pass on S1 has risen from 175 to 194 -- you can see the trends here: A peek inside the USMLE sausage factory: setting the Step 1 minimum passing score). I think it's the former.

So, problems with this study:
1. If 215 => 93% board pass, that's a 215 in 1997 which was the average. It certainly isn't a 215 today.
2. A 93% board pass rate would put an IM program in the 3rd or 4th quartile (bottom) of IM programs. That is a terrible board pass rate.

So it's an interesting study, but limited.


This is one of the options listed. Would students be required to take both? Or could they just take S1 and if "happy", just plan to take S2 later (much like today)? And since S2 scores are higher than S1 scores on average, this will just lead to further score inflation.

It's a tradeoff, and they are trying to have it both ways. Since CS is P/F, if you fail you can't show how badly you failed (just barely vs miserably), and then when you subsequently pass can't show whether you passed barely or by a mile. Adding scoring would help that, but then we're back to S1 mania issues with scores.

It's an enormous problem. Everyone is in the top 5% in LOR's. Many MSPE's are at least more reasonable, but some continue to place students into "groups" but then don't tell you how many are in each group.



Either is a reasonable option. I agree the first will drive students crazy trying to determine what number of applications is "best". And the latter, some fields may decide to only review the preference apps, placing a defacto hard limit on applications and forcing students to decide which programs they are really competitive for.

The explorer tool may end up being useful, probably better than where we are now.

Sure, but then your entire career hangs in the balance of a single evaluation. In the ED you work with a mix of people so perhaps it's a bit more fair. And schools would need to ensure that not 95% of student get the top rating. Soon students will be complaining that the evaluations are subjective and unfair, and that they want an objective system. Plus this create a situation where you only get your eval when your training is just about done -- if it's poor, you have little time to reboot your career.

And, not useful for IMG's. So really not a system wide answer.

Might be taking your comment out of context (sorry!), but this is also a common statement -- that because the SD is wide enough that most scores can't be compared. This is not exactly accurate. First, these statistics are used to compare the averages of two groups to see if they come from the same population, not comparing two individual scores. Second, this all assumes a 95% confidence interval -- I'm happy with a much lower level of confidence. The number quoted (16?) - not sure where that is from. If it's the standard dev of the S1 score distribution, that's not what you want.

My bottom line:
1. If the FSMB and the AAMC want to change to P/F, there's no one to stop them.
2. As they seem to have punted this to some still-to-be-determined group of concerned citizens, I doubt they will be able to come to any consensus.
3. As a non trusting sort, I would move ahead with development of a specialty specific exam so that, as a field, we simply don't need to worry about this any more. Likely outcome is --> group formed --> lots of talk --> no agreement --> AAMC/NBME/FSMB decide whatever they want anyway.

Thank you very much for all that insights! I am looking forward to your next post about #3 (maybe even one exam for all specialties, a new "S1", like you said last time)!

 

[Lucca]

Went away for the weekend. Lots to get caught up on.



If P/F for S1 were to come to pass, programs would need to make decisions using other factors. S2 might be used, forcing students to take it earlier. I don't think we're ever going back to a world where there is no standardized exam to take. The big fields will make their own, if that's what's needed.



There is no clear timeline. The "plan" is to convene some sort of a group to review all of these issues, and (theoretically) make a decision in 2019-2020. But getting consensus is going to be difficult, if not impossible. Even if a decision is made in this timeframe, any change would be delayed for a few years. This isn't worth worrying about.



Answered by some others already, but depends on what you mean by "holistically". MSPE's do stratify students into groups. LOR's are not very helpful -- all say the student is in the top 5%, and relying on LOR's becomes a "who you know" / "How connected you are" game. So all we're left with are clinical grades and MSPE.

And, that's for US grads. 50% of IM spots are filled by IMG's, and many of them have useless MSPE's. All they have are their USMLE scores.


One of the options was to divide into quartiles or quintiles. It would be fine as far as I'm concerned. Problem is, if you're in that bottom group, I can't tell if you just barely passed, or if your performance was better than that. Some programs will assume the worst.

We've had this discussion before, but 1) that might not be possible, and 2) this just elevates the importance of other parts of the application which may be just as flawed, or worse. Plus, IMG's fill a large number of spots in the US, and their applications may be mostly worthless except for exam scores. And if we just switch to S2 scores, we're just pushing the problem onto a new exam (and students will want "dedicated" 8 weeks to study for it).

This study often comes up in this discussion. It was done looking at graduates in 1997-2000, when step scores were lower (av S1 was 215). The USMLE "states" that scores are equivalent over time. Either that's wrong and there's some inflation over time, or we licensed a bunch of people in the past who should have failed (as the minimum pass on S1 has risen from 175 to 194 -- you can see the trends here: A peek inside the USMLE sausage factory: setting the Step 1 minimum passing score). I think it's the former.

So, problems with this study:
1. If 215 => 93% board pass, that's a 215 in 1997 which was the average. It certainly isn't a 215 today.
2. A 93% board pass rate would put an IM program in the 3rd or 4th quartile (bottom) of IM programs. That is a terrible board pass rate.

So it's an interesting study, but limited.


This is one of the options listed. Would students be required to take both? Or could they just take S1 and if "happy", just plan to take S2 later (much like today)? And since S2 scores are higher than S1 scores on average, this will just lead to further score inflation.

It's a tradeoff, and they are trying to have it both ways. Since CS is P/F, if you fail you can't show how badly you failed (just barely vs miserably), and then when you subsequently pass can't show whether you passed barely or by a mile. Adding scoring would help that, but then we're back to S1 mania issues with scores.

It's an enormous problem. Everyone is in the top 5% in LOR's. Many MSPE's are at least more reasonable, but some continue to place students into "groups" but then don't tell you how many are in each group.



Either is a reasonable option. I agree the first will drive students crazy trying to determine what number of applications is "best". And the latter, some fields may decide to only review the preference apps, placing a defacto hard limit on applications and forcing students to decide which programs they are really competitive for.

The explorer tool may end up being useful, probably better than where we are now.

Sure, but then your entire career hangs in the balance of a single evaluation. In the ED you work with a mix of people so perhaps it's a bit more fair. And schools would need to ensure that not 95% of student get the top rating. Soon students will be complaining that the evaluations are subjective and unfair, and that they want an objective system. Plus this create a situation where you only get your eval when your training is just about done -- if it's poor, you have little time to reboot your career.

And, not useful for IMG's. So really not a system wide answer.

Might be taking your comment out of context (sorry!), but this is also a common statement -- that because the SD is wide enough that most scores can't be compared. This is not exactly accurate. First, these statistics are used to compare the averages of two groups to see if they come from the same population, not comparing two individual scores. Second, this all assumes a 95% confidence interval -- I'm happy with a much lower level of confidence. The number quoted (16?) - not sure where that is from. If it's the standard dev of the S1 score distribution, that's not what you want.

My bottom line:
1. If the FSMB and the AAMC want to change to P/F, there's no one to stop them.
2. As they seem to have punted this to some still-to-be-determined group of concerned citizens, I doubt they will be able to come to any consensus.
3. As a non trusting sort, I would move ahead with development of a specialty specific exam so that, as a field, we simply don't need to worry about this any more. Likely outcome is --> group formed --> lots of talk --> no agreement --> AAMC/NBME/FSMB decide whatever they want anyway.

Thank you for the detailed reply.

 

[JooceMan137]

As someone who has now taken Step 1 my opinion has not changed. It is very clinically relevant, as most of my questions pertained to pharm, pathophys, path, phys, understanding basic biostats, very clinically relevant anatomy, basic next step management, etc. I still have yet to see an argument for a change in the current model that holds any water, let alone a good reason for this test to go P/F.

Change for the sake of change instead of the right change, will have devastating unforeseen consequences for large swaths of medical students.

Indeed. Although Step 1 does test on nitty gritty cold science type stuff (DNA synthesis, etc.), it also tests heavily on figuring out a diagnosis based on clinical symptoms. I mean, what good is a physician who can't recognize that a patient coming in with jaundice, early satiety, quick weight loss, and a history of smoking should get some imaging done on the abdomen to rule out pancreatic cancer?

 

[TheRealCookieMonster]

AAMC/NBME/FSMB decide whatever they want anyway.

Ahhh sweet futility...

 

[afib123]

As someone who has now taken Step 1 my opinion has not changed. It is very clinically relevant, as most of my questions pertained to pharm, pathophys, path, phys, understanding basic biostats, very clinically relevant anatomy, basic next step management, etc. I still have yet to see an argument for a change in the current model that holds any water, let alone a good reason for this test to go P/F.

Change for the sake of change instead of the right change, will have devastating unforeseen consequences for large swaths of medical students.

The extremely simple solution is to tweak step 1 and take out the 20% basic science questions and make those clinical questions too. I mean step 1 is an excellent exam at the ideal time that just needs some tweaking.

I think step 1 is more clinically relevant than the curriculum for the first 2 years that have unnecessary basic science classes taught by PhDs. Make the curriculum clinical. Make step 1 clinical.

The first problem is the curriculum taught in medical schools is extremely outdated. I wouldn't be surprised if people using boards and beyond and pathoma and sketchy to study are doing better on the wards than the people that use the school's curriculum. Those resources were more clinically relevant than the 3 hours of lectures each that day were drawn out with clinically irrelevant stuff.

More than step 1, the clinical curriculum needs to be changed. Students are becoming horrible in their physical exam skills. People need to start seeing patients earlier. The end of cardiology module should be followed a few weeks seeing patients on the cardiology service. The cardiology module should be prepping students to become excellent at recognizing murmurs, assessing volume status, learning how to manage chest pain, learning to auscultate S3, S4, friction rub, measuring JVD, dosing HF meds, HF pathophysiology. Instead most people across the country spend wasted time on wiggers diagrams, flow volume loops, etc. Those are helpful sure but should not be hours and hours of lecture. The clinical stuff should be what's emphasized.

Step 1 actually tests more of the clinically relevant stuff than most school tests. Change the curriculum. Change step 1. Not make it P/F.

 

[Lawpy]

As someone who has now taken Step 1 my opinion has not changed. It is very clinically relevant, as most of my questions pertained to pharm, pathophys, path, phys, understanding basic biostats, very clinically relevant anatomy, basic next step management, etc. I still have yet to see an argument for a change in the current model that holds any water, let alone a good reason for this test to go P/F.

Change for the sake of change instead of the right change, will have devastating unforeseen consequences for large swaths of medical students.

The extremely simple solution is to tweak step 1 and take out the 20% basic science questions and make those clinical questions too. I mean step 1 is an excellent exam at the ideal time that just needs some tweaking.

I think step 1 is more clinically relevant than the curriculum for the first 2 years that have unnecessary basic science classes taught by PhDs. Make the curriculum clinical. Make step 1 clinical.

I'm happy Step 1 is clinically relevant because some of the faculty here complain it's so massively heavy on basic science

 

[afib123]

I'm happy Step 1 is clinically relevant because some of the faculty here complain it's so massively heavy on basic science

There is no doubt there is basic science on there. But the medical school curriculum itself is heavily basic science and the step 1 basic science emphasis is less than the basic science in my school's curriculum. If I had to compare my school's curriculum and in house exams to step 1, hands down step 1 is more clinically relevant.
Take biochemistry and gross anatomy for example. Step 1 tests you on mostly clinically relevant biochemistry and anatomy. On the other hand, you spend (in a non-organ system based curriculum) at least 3 months on gross anatomy and biochemistry (why the heck is that stuff being taught for such a long period of time?).
Medical school curriculum is what needs a revamp. Can be much much more efficient and clinically relevant. Refer to my previous post on how I think the curriculum can be changed. None of this problem based learning etc. It should change in its content, not the way of delivery.

I worked under a foreign physician last 2 weeks and I am just amazed by his physical exam skills and clinical acumen. The new US attendings that I worked with are far behind in those skills. Why? Because we waste time on useless physiology and basic science crap. The foreign guys start seeing tons of patients extremely early.

Step 1 has some of the basic science too but I think I am a better clinical student from studying for step 1 than I would be if I studied for in house exams. I would be an even better student if the curriculum and step 1 are changed.

It is a different topic altogether, but when you finally come to clinical year they start evaluating you in a BS manner which makes people wary to ask "stupid" questions when the entire point is to learn. Med students start copying their residents' notes and presentations to look good on rounds. I would much rather look stupid on rounds, assess and present my patients on my own without looking or talking to anyone about them. Learn from my mistakes. But because of the grade fear, I start taking the safe route of seeing less patients, going over the plan with the residents, and leaving home for a stupid shelf exam.

I think shelf exams and step 2 and third year grades are what should be removed. Third year should be to see patients. Students be encouraged to spend more time on wards. See 5-6 patients each day. Go home and learn from the patients. But instead you pick 2-3 patients, present them, get out early and study for a stupid shelf so you can get into a good residency.

You have the entire first 2 years to study for a test. Study for it. Take it. Then immerse yourself in the wards is how I would structure medical school. I thought I was a good student until the last few weeks when I started to work with a smart attending.

Can you believe that an orthopedic intern paged the medicine service for diabetes management?! I mean that is pitiful if you don't know how to dose simple insulin as a doctor.

 

[Med Ed]

The extremely simple solution is to tweak step 1 and take out the 20% basic science questions and make those clinical questions too. I mean step 1 is an excellent exam at the ideal time that just needs some tweaking.

Step 1 is far more than 20% basic sciences. The dictum from the NBME is to abandon straight recall questions and instead frame every single item around a clinical vignette or experimental scenario. Essentially they are embedding a basic science exam in questions that appear clinically focused. That's not to say that Step 1 does not contain bona fide clinical questions. The NMBE has been deliberate in making Step 1 a bit more clinical than it used to be, while trying to infuse some basic science content into Step 2. In essence making the two exams a little more alike than in years past. In some ways this reflect the fact that the old school 2 years of class and 2 years in the hospital isn't how medical school is done anymore.

Make the curriculum clinical. Make step 1 clinical.

Remove or marginalize the basic sciences too much and congrats, you just created a really long and expensive DNP program.

More than step 1, the clinical curriculum needs to be changed. Students are becoming horrible in their physical exam skills. People need to start seeing patients earlier. The end of cardiology module should be followed a few weeks seeing patients on the cardiology service. The cardiology module should be prepping students to become excellent at recognizing murmurs, assessing volume status, learning how to manage chest pain, learning to auscultate S3, S4, friction rub, measuring JVD, dosing HF meds, HF pathophysiology. Instead most people across the country spend wasted time on wiggers diagrams, flow volume loops, etc. Those are helpful sure but should not be hours and hours of lecture. The clinical stuff should be what's emphasized.

Students don't need to see patients earlier. Practically every allopathic school in the country now has "early clinical exposure" with preceptorships or other experiences. If physical exam skills are deteriorating the reason is actually Step 1 itself. Getting students interested in anything other than Step 1 for the first two years of medical school has become a lost cause.

"Physical examination?"
"Step 1."
"Critical analysis?"
"Step 1."
"Interpersonal skills?"
"Step 1."
"Clinical reasoning?"
"Step 1."

And so on.

If you're not directly involved in UME then it's understandably difficult to appreciate the situation, but basically over the past 5 years the entire system has gone into core meltdown. The students currently inside the machine obviously lack perspective, and cannot see that while Step 1 has been an important milestone since its inception, things weren't always like this. Medical students in the past took for granted the luxury of developing themselves across multiple domains in a manner that currently seems quaint.