If you were a PD who just found out Step 1 became P/F, what metric would you use to replace it?

This forum made possible through the generous support of SDN members, donors, and sponsors. Thank you.
Hmmmmm how would I evaluate applicants without a single test score that shows no correlation with residency performance and where the scaling error alone means there’s no difference between a 230 and a 245??

Grades, shelves, letters, research, CK, away rotation performance, etc. It will be more time consuming on the front end for some that screen but should mean little difference for those that don’t. I’m trying to think of the last time someone’s step one score came up during our rank meeting and I can’t remember one. We end up talking about their research and story and especially letters and personal discussions with people we know.

So yes more struggle on the front end especially for big programs that get many applications, but should be pretty minor.
 
operaman said:
Hmmmmm how would I evaluate applicants without a single test score that shows no correlation with residency performance and where the scaling error alone means there’s no difference between a 230 and a 245??

Grades, shelves, letters, research, CK, away rotation performance, etc. It will be more time consuming on the front end for some that screen but should mean little difference for those that don’t. I’m trying to think of the last time someone’s step one score came up during our rank meeting and I can’t remember one. We end up talking about their research and story and especially letters and personal discussions with people we know.

So yes more struggle on the front end especially for big programs that get many applications, but should be pretty minor.
Click to expand...


Is it true that someone going to a low tier MD is basically screwed now out of a top residency spot since it will be much harder for them to differentiate themselves over the people at T20s who want those spots (and will now likely get them)?
 
operaman said:
Hmmmmm how would I evaluate applicants without a single test score that shows no correlation with residency performance and where the scaling error alone means there’s no difference between a 230 and a 245??

Grades, shelves, letters, research, CK, away rotation performance, etc. It will be more time consuming on the front end for some that screen but should mean little difference for those that don’t. I’m trying to think of the last time someone’s step one score came up during our rank meeting and I can’t remember one. We end up talking about their research and story and especially letters and personal discussions with people we know.

So yes more struggle on the front end especially for big programs that get many applications, but should be pretty minor.
Click to expand...
At rank meetings academics may be less important, but getting the interview is necessary too. Being in ENT at a good program leads me to believe that most, if not all, of your interviewees are excellent academic candidates or they wouldnt be there. Your post probably isnt generalizable to other fields.
 
I'll begin this response by noting that I agree that the effective use of USMLE exams as a standardized comparison tool was never how the exams were intended to be applied. Officially it is a licensing exam, setting a bare minimum standard to become a licensed physcian and nothing more. I wish there was something better that was still universal to all applicants. Maybe we need to start a new exam - the "Graduate Medical Education Aptitude Test" - like the SAT / MCAT but for residency application... like a combination of medical knowledge, critical thinking skills, and emotional intelligence testing, and designed specifically to be used to compare applicants. (that is, of course, /s...but my joke proves a point regarding the challenges of evaluating applicants).

Definitely Step 2 CK will replace as the initial screening tool. When you have >1000 applications there is simply no realistic way to carefully examine every application in depth. Shelf scores, or even the MCAT, could be looked at more heavily, I suppose. In the past we've never really paid attention to those. But there needs to be some objective metric - something that all applicants have in common external to their individual schools - that helps standardize students against one another. USMLE exams are definitely correlated with ability to pass board exams, at least above a certain threshold. A 235 vs 260 actually wasn't given as much weight due to these diminishing returns. But a 215 or below applicant, no matter how much we loved them otherwise, is definitely considered "high risk" for academic difficulty in residency.

Letters, personal statements, and even third year clerkship grades (thanks to wild distribution variability between schools) are honestly not very helpful in this initial screening. You can thank grade inflation, P/F schools, and the explosion of new medical schools of dubious quality for the erosion of helpfulness of clerkship grades. Once we've narrowed our applicant pool down to the 130-150 we actually want to interview then we can more thoroughly dive into those more subjective things. But after having read all the applications, PS's, and LoRs this last cycle for applicants we interviewed, these are frankly really hard to evaluate. It's kind of a game. They all sound the same. Maybe 10% of letters or PS's really stand out as "something special", and maybe another 10% stand out as unusually weak (letter writers have a special code language they use to gently warn us about applicants they don't love but are politely writing a letter for). But 80% of them are basically about the same. The interview itself is not particularly helpful for most applicants either, again maybe weeding out 1/10th who come across poorly in person, and helping another 1/10th stand out. Unfortunately there is no really great way to confidently evaluate that many applicants based on what's in their application alone. It's kind of just a combination of all the above plus subjective "gut feelings" from everyone who interacted with them or is on the rank list committee. The only time we are ever solidly confident about applicants is when they are either students from our own medical school or did an away rotation with us.
 
Morzh said:
I'll begin this response by noting that I agree that the effective use of USMLE exams as a standardized comparison tool was never how the exams were intended to be applied. Officially it is a licensing exam, setting a bare minimum standard to become a licensed physcian and nothing more. I wish there was something better that was still universal to all applicants. Maybe we need to start a new exam - the "Graduate Medical Education Aptitude Test" - like the SAT / MCAT but for residency application... like a combination of medical knowledge, critical thinking skills, and emotional intelligence testing, and designed specifically to be used to compare applicants. (that is, of course, /s...but my joke proves a point regarding the challenges of evaluating applicants).

Definitely Step 2 CK will replace as the initial screening tool. When you have >1000 applications there is simply no realistic way to carefully examine every application in depth. Shelf scores, or even the MCAT, could be looked at more heavily, I suppose. In the past we've never really paid attention to those. But there needs to be some objective metric - something that all applicants have in common external to their individual schools - that helps standardize students against one another. USMLE exams are definitely correlated with ability to pass board exams, at least above a certain threshold. A 235 vs 260 actually wasn't given as much weight due to these diminishing returns. But a 215 or below applicant, no matter how much we loved them otherwise, is definitely considered "high risk" for academic difficulty in residency.

Letters, personal statements, and even third year clerkship grades (thanks to wild distribution variability between schools) are honestly not very helpful in this initial screening. You can thank grade inflation, P/F schools, and the explosion of new medical schools of dubious quality for the erosion of helpfulness of clerkship grades. Once we've narrowed our applicant pool down to the 130-150 we actually want to interview then we can more thoroughly dive into those more subjective things. But after having read all the applications, PS's, and LoRs this last cycle for applicants we interviewed, these are frankly really hard to evaluate. It's kind of a game. They all sound the same. Maybe 10% of letters or PS's really stand out as "something special", and maybe another 10% stand out as unusually weak (letter writers have a special code language they use to gently warn us about applicants they don't love but are politely writing a letter for). But 80% of them are basically about the same. The interview itself is not particularly helpful for most applicants either, again maybe weeding out 1/10th who come across poorly in person, and helping another 1/10th stand out. Unfortunately there is no really great way to confidently evaluate that many applicants based on what's in their application alone. It's kind of just a combination of all the above plus subjective "gut feelings" from everyone who interacted with them or is on the rank list committee. The only time we are ever solidly confident about applicants is when they are either students from our own medical school or did an away rotation with us.
Click to expand...
How refreshing it is to hear from someone with actual knowledge and who is not blindly hyperventilating! Many thanks, colleague!
 
operaman said:
Hmmmmm how would I evaluate applicants without a single test score that shows no correlation with residency performance and where the scaling error alone means there’s no difference between a 230 and a 245??

Grades, shelves, letters, research, CK, away rotation performance, etc. It will be more time consuming on the front end for some that screen but should mean little difference for those that don’t. I’m trying to think of the last time someone’s step one score came up during our rank meeting and I can’t remember one. We end up talking about their research and story and especially letters and personal discussions with people we know.

So yes more struggle on the front end especially for big programs that get many applications, but should be pretty minor.
Click to expand...

Step 1 score affects whether you get the interview. If you dont have that, suddenly you can't even talk about the story or personality before you start eliminating people.

There is a certain subconscious wow factor that primes people to talk positively about someone's intelligence once they see a high score. You may not mention it, but the knowledge that someone scores high on step 1 will change slightly the tone of the conversation.
 
I think the question assumes there needs to be a singular metric to judge applicants to begin with. Step 1 wasn't supposed to be a metric in the first place, its a board exam. No employer looks or wants to look at a single metric to decide if they're going to hire someone or not. At least I don't. The long short of it is the AAMC is screwing over residency programs and applicants.

I imagine from the perspective PD's and adcoms, there are a handful of applicants where they're like "Yeah, we want this person.", a few that they definitely don't want, but most fall into the category of "fine, but I don't have a real preference". We suffer from the same problem where I work, however we don't get thousands of applications mostly from people who would go 20 other places before coming here. I think PD's have this problem. So what's a good way to filter everyone? Numbers.

So the problem comes from the way that ERAS is designed. If you want to apply for a job, you can't just apply to every company ever with a click of a button. There are different job posting websites, you have to make cover letters for each company, it's just logistically much harder to do. Now take ERAS, people put in their junk, check boxes, overdraft their bank account and apply to 60 programs. So you've taken a group of highly competitive, highly risk-averse people the ability to send limitless amounts of applications. They're going to assume that if they send more applications than average because they assume that will give them an above average change to match. Now think about that...

The trend you get is that with each generation of pokemon, the average applicant sends about 7 more applications. So where I think we're at now is PD's have to get a PhD in astrophysics to figure out of the 3,000 people who applied to their program, how many of them would rank them highly enough to be worth their time. When in reality they should just be getting a sane amount of applications from the start from people that actually want to go to their program.

Matchmaking systems don't really work when have everyone is ranking everyone. If you're asked to list your preferences of a sufficient number of things, there will come a point where your preferences are no longer rational. The entire system is designed to match you with the programs/applicants you like best, it defeats the purpose if people are applying to and ranking programs they don't really care for.

There has to be strict application limits, fees will not stop people from applying to an ever-increasing number of programs. Statistically most medical students come from very wealthy families, money is no object. Furthermore, if you're already 200K in the hole, was 20k extra debt really gonna stop you? Asking med students to stop sending so many applications is essentially telling them not to worry. It's not going to work, and you look silly for trying.

This will force applicants to be more thoughtful about where they apply, and free programs to actually look beyond a three digit score.
 
dienekes88 said:
Step 1 score affects whether you get the interview. If you dont have that, suddenly you can't even talk about the story or personality before you start eliminating people.

There is a certain subconscious wow factor that primes people to talk positively about someone's intelligence once they see a high score. You may not mention it, but the knowledge that someone scores high on step 1 will change slightly the tone of the conversation.
Click to expand...

every program is different but at least at ours I think many people would be surprised at how little impact step scores have. We’ve been chatting about this recently since the score change came out and quite a few people have admitted they don’t even look at step scores when doing the first pass. As more people have become aware of how bad the test is at discerning between applicants and how meaningless it is as a predictor of success, we are all caring about it less and less.

Obviously some of this is our program since we get generally stellar applicants overall and a relatively small pool so we have that luxury. An IM or peds program that gets thousands of apps probably has to do more broad screening simply to manage the workload.

I think you can also gauge someone’s overall ability by their overall application. Someone who has the chops to score a 90+ percentile step score likely has the chops to do well academically while also juggling research and interesting ECs. Someone struggling to hit that 50th percentile probably doesn’t have the available bandwidth to do all the other things that make an app stand out. The knowledge base of a strong step score likely translates into clinical grades too. The junior AOA folks in my class all of whom were in the 260+ range all tended to be strong on the wards as well. While there may be exceptions out there, I think a wholistic view of an application can probably tell me plenty.
 
operaman said:
Hmmmmm how would I evaluate applicants without a single test score that shows no correlation with residency performance and where the scaling error alone means there’s no difference between a 230 and a 245??

Grades, shelves, letters, research, CK, away rotation performance, etc. It will be more time consuming on the front end for some that screen but should mean little difference for those that don’t. I’m trying to think of the last time someone’s step one score came up during our rank meeting and I can’t remember one. We end up talking about their research and story and especially letters and personal discussions with people we know.

So yes more struggle on the front end especially for big programs that get many applications, but should be pretty minor.
Click to expand...

Meanwhile you wouldn't have even been talking about those students at rank meetings who scored 198 on the Step 1 to begin with. But hey, they got their name on two papers and had grade inflation at their institution.
 
walter_heisenberg said:
Is it true that someone going to a low tier MD is basically screwed now out of a top residency spot since it will be much harder for them to differentiate themselves over the people at T20s who want those spots (and will now likely get them)?
Click to expand...
Yeah
 
operaman said:
every program is different but at least at ours I think many people would be surprised at how little impact step scores have. We’ve been chatting about this recently since the score change came out and quite a few people have admitted they don’t even look at step scores when doing the first pass. As more people have become aware of how bad the test is at discerning between applicants and how meaningless it is as a predictor of success, we are all caring about it less and less.

Obviously some of this is our program since we get generally stellar applicants overall and a relatively small pool so we have that luxury. An IM or peds program that gets thousands of apps probably has to do more broad screening simply to manage the workload.

I think you can also gauge someone’s overall ability by their overall application. Someone who has the chops to score a 90+ percentile step score likely has the chops to do well academically while also juggling research and interesting ECs. Someone struggling to hit that 50th percentile probably doesn’t have the available bandwidth to do all the other things that make an app stand out. The knowledge base of a strong step score likely translates into clinical grades too. The junior AOA folks in my class all of whom were in the 260+ range all tended to be strong on the wards as well. While there may be exceptions out there, I think a wholistic view of an application can probably tell me plenty.
Click to expand...

Speaking in generalities, you are mostly right. But it's not always easy to compare the strong (but doesn't walk on water) at a smaller school to the middle achiever at a much larger school. It's the age-old story that is the same as applying to medical school.

It'd be great if every applicant was so easy to sort out. Junior AOA mega stud with 4 first author publications vs. The kid with weak LORs and no pubs. Most of the students are somewhere in between.
 
They do a token system in SOAP so applicants don’t apply to every available program. Why can’t this be something done in the main match? Or do what some specialties do and require secondaries for each program.
 
Phloston said:
Meanwhile you wouldn't have even been talking about those students at rank meetings who scored 198 on the Step 1 to begin with. But hey, they got their name on two papers and had grade inflation at their institution.
Click to expand...

I think the mythical star student who is just bad at tests and lands a 198 is largely that: a myth. We’ve definitely matched applicants with step scores far below what you’d expect, though to my knowledge never that low.

But someone whose best effort lands them a 198 is probably not coming at us with an amazing CV. I’ve tutored a number of struggling students through the years and someone on the edge like that is just trying to pass and graduate. They aren’t banging out pubs and schmoozing their home department because all their free time is focused on just passing the next marker. They aren’t the people who end up applying for the most competitive programs in the most competitive fields.
 
US grad, letters, personal statements. I assume Step 2 will also end up pass/fail once the NBME realizes programs are trying to subvert their Step 1 changes by using Step 2.
 
Mythoclast said:
So the problem comes from the way that ERAS is designed. If you want to apply for a job, you can't just apply to every company ever with a click of a button. There are different job posting websites, you have to make cover letters for each company, it's just logistically much harder to do. Now take ERAS, people put in their junk, check boxes, overdraft their bank account and apply to 60 programs. So you've taken a group of highly competitive, highly risk-averse people the ability to send limitless amounts of applications. They're going to assume that if they send more applications than average because they assume that will give them an above average change to match. Now think about that...

The trend you get is that with each generation of pokemon, the average applicant sends about 7 more applications. So where I think we're at now is PD's have to get a PhD in astrophysics to figure out of the 3,000 people who applied to their program, how many of them would rank them highly enough to be worth their time. When in reality they should just be getting a sane amount of applications from the start from people that actually want to go to their program.

Matchmaking systems don't really work when have everyone is ranking everyone. If you're asked to list your preferences of a sufficient number of things, there will come a point where your preferences are no longer rational. The entire system is designed to match you with the programs/applicants you like best, it defeats the purpose if people are applying to and ranking programs they don't really care for.

There has to be strict application limits, fees will not stop people from applying to an ever-increasing number of programs. Statistically most medical students come from very wealthy families, money is no object. Furthermore, if you're already 200K in the hole, was 20k extra debt really gonna stop you? Asking med students to stop sending so many applications is essentially telling them not to worry. It's not going to work, and you look silly for trying.

This will force applicants to be more thoughtful about where they apply, and free programs to actually look beyond a three digit score.
Click to expand...

I think it would be reasonable to put in place hard limits to applications and potentially interviews attended. However, I don't know how one would even go about setting these limits for different fields, as they all seem like completely different worlds. And what about couples matchers? I also think that considerably more information and transparency would be needed from the programs so that applicants could actually figure out what a "target program" is for them. I used FREIDA and residency explorer when applying which are great tools in theory but both had outdated data (like, Step 1 cutoffs being listed as previous passing scores that are now failing scores). Residency Explorer is difficult to use practically, it compares you to applicants who matched to each program from 2014-2018 which is great because you can definitely see if you would be an extreme outlier one way or another at any program. But it doesn't show trends and if you yourself are an average applicant in terms of scores and quantity of experiences, you are just going to see a lot of "middle 50% of matched applicants" which doesn't help you narrow down which programs to apply to.

Maybe one of the best ways of imposing limitations would be a brief "why us" prompt at each program. The only additional cost is time. So many people are writing letters of interest anyway, why not standardize it and make it the first screening tool? It would allow applicants to write about specific professional interests and location preferences without having to write completely separate personal statements. I'm not sure if anyone would actually like this idea but I do think it would impose limits in a fair way.
 
I disagree with using aptitude test at the med school stage. I am good at them, so I am actually biased towards them but even then I don't think that's a good idea. Medicine is generally not a creative intuitive job for deep thinkers. It is a technical craft requiring long hours and a massive knowledge base. Some exceptional doctors do add to the fund of knowledge via device innovation, ground breaking research, etc. But residency is for becoming a competent clinician. A heavily knowledge based not so gloaded exam like the Step 1 is a good marker for work ethic and willingness to accumulate the massive knowledge base to become a good clinician.

But I can see why the more relevant Step2ck is better suited. I would hope this change leads to 1 year preclinical curriculum, where some of the excess basic science is trimmed. Then slow down clinical rotations to span say 1.5 years. Have shelf exam scores reported and CK reported for them. I wouldn't also mind mandating preclinical shelf taking and reporting. Have two versions: one systems based and one traditional.

This gives a big data set. Also have quintiles so people don't go ocpd for every point. And don't report to students the exact score, just the quintile so they won't even know how close or far they were. And especially don't allow exact score release to programs, only quintiles.

Programs can have their own matrices on how to wvaluate quintiles on different exams (ie. Surgery ones may value anatomy shelf grades higher than medicine ones on average).
 
I've done a little residency selection screening in a past life so if I were a PD I would take a deep dive into the LORs, including the backgrounds of the letter writers. I'd be looking for how long the attending has been in practice, whether or not the attending knew the candidate for more than a day, if the attending speaks to how residents and staff like the candidate, specific examples of professionalism and potential etc. It would definitely be time consuming, but with 2-3 other reviewers who are trained to look for what I want, it could definitely get done in a reasonable amount of time.
 
Oedipa Maas said:
I think it would be reasonable to put in place hard limits to applications and potentially interviews attended. However, I don't know how one would even go about setting these limits for different fields, as they all seem like completely different worlds. And what about couples matchers? I also think that considerably more information and transparency would be needed from the programs so that applicants could actually figure out what a "target program" is for them. I used FREIDA and residency explorer when applying which are great tools in theory but both had outdated data (like, Step 1 cutoffs being listed as previous passing scores that are now failing scores). Residency Explorer is difficult to use practically, it compares you to applicants who matched to each program from 2014-2018 which is great because you can definitely see if you would be an extreme outlier one way or another at any program. But it doesn't show trends and if you yourself are an average applicant in terms of scores and quantity of experiences, you are just going to see a lot of "middle 50% of matched applicants" which doesn't help you narrow down which programs to apply to.
Click to expand...


I don't think it's really necessary to worry about. Realistically, the vast majority of people are going to match into their preferred specialty, unless they want to do orthopedic surgery increasing the number of applications won't fix that problem. There simply aren't enough spots for people, I know it sounds completely crazy, but that's what application fever is. If a lot of people go unmatched, they will just have to scramble. I suppose the one thing you could do is show the programs that have a low number of applicants. If the application limit caused there to be too many people unmatched they could just do a second match cycle over the month or just have a big scramble. There are lots of things that could be done to prevent such an outcome in the future.


Oedipa Maas said:
Maybe one of the best ways of imposing limitations would be a brief "why us" prompt at each program. The only additional cost is time. So many people are writing letters of interest anyway, why not standardize it and make it the first screening tool? It would allow applicants to write about specific professional interests and location preferences without having to write completely separate personal statements. I'm not sure if anyone would actually like this idea but I do think it would impose limits in a fair way.
Click to expand...


That works only in the case where one program has a prompt and the others don't. There was a study where one ortho program had a prompt and only like 30% of applicants actually did it, lul. The problem with prompts is that when every program has a prompt, nobody has a prompt. The more prompts you do the more likely you are to see the exact same one in the future. It might work for a time, but then you will have people pre-writing prompts for programs in M4 downtime and 5 years down the road we're right back in the same spot. Hard limits on applications are the most fair/good way to do fix the problem, full stop.
 
I have sat on resident selection comittees and my number one metric is have we had this person on our service and can I teach them? That's my job. We know if they are hard workers and how they interact with faculty, residents and staff. Some really bright residents have been unteachable. Never rotated with us but had good scores, letters, and good schools. Scores will get you noticed, but auditioning allows us to see the whole package,
 
Last edited:
If everyone agrees that a passing score is all that is needed to become a doc in any field, and the pass rate is > 90% then why (other than $$$), doe the organisations insist on putting students through that.

With such a high pass rate is it really a weed out test?
 
FutureInternist said:
If everyone agrees that a passing score is all that is needed to become a doc in any field, and the pass rate is > 90% then why (other than $$$), doe the organisations insist on putting students through that.

With such a high pass rate is it really a weed out test?
Click to expand...

Most of the reason is due to the apps to spot ratio. The amount of apps makes it impossible for programs to do h*listic review. Hate that word, lol. They have to decrease the number somehow in order to make app review reasonable. Step 1 has been used to do this for decades. Now it'll be step 2. Limiting apps is another, more direct way to deal with the issue.
 
Angus Avagadro said:
I have sat on resident selection comittees and my number one metric is have we had this person on or service and can I teach them? That's my job. We know if they are hard workers and how they interact with faculty, residents and staff. Some really bright residents have been unteachable. Never rotated with us but had good scores, letters, and good schools. Scores will get you noticed, but auditioning allows us to see the whole package,
Click to expand...

^Valuable post here kids.
 
Angus Avagadro said:
I have sat on resident selection comittees and my number one metric is have we had this person on or service and can I teach them? That's my job. We know if they are hard workers and how they interact with faculty, residents and staff. Some really bright residents have been unteachable. Never rotated with us but had good scores, letters, and good schools. Scores will get you noticed, but auditioning allows us to see the whole package,
Click to expand...

I think around 50% of our residents are either home students or great rotators. Our rank lists typically have the well known students listed very high. There’s just no substitute for the personal knowledge of how someone works.

Granted we all know they’re in audition mode for the month so they are probably showing something better than their average effort, but at least we know what’s possible. That said, we probably offer interviews to 10-20% of rotators because many people manage to slip out of audition mode pretty quickly.

Side note: I’ve looked back at rotators we didn’t like and most all of them have gone on to be successful residents in other programs.
 
operaman said:
I think around 50% of our residents are either home students or great rotators. Our rank lists typically have the well known students listed very high. There’s just no substitute for the personal knowledge of how someone works.

Granted we all know they’re in audition mode for the month so they are probably showing something better than their average effort, but at least we know what’s possible. That said, we probably offer interviews to 10-20% of rotators because many people manage to slip out of audition mode pretty quickly.

Side note: I’ve looked back at rotators we didn’t like and most all of them have gone on to be successful residents in other programs.
Click to expand...
could you expound on this a little bit. I was under the impression the residents and faculty dont want to send invites to robots either who are constantly in interview mode for the entire month as well.
 
Angus Avagadro said:
I have sat on resident selection comittees and my number one metric is have we had this person on our service and can I teach them? That's my job. We know if they are hard workers and how they interact with faculty, residents and staff. Some really bright residents have been unteachable. Never rotated with us but had good scores, letters, and good schools. Scores will get you noticed, but auditioning allows us to see the whole package,
Click to expand...
Away rotations should be against LCME rules. Why should wealthier kids be able to pay money to rotate at a top program to subsequently be ranked higher?

It almost sounds like a politician buying an election.
 
operaman said:
I think around 50% of our residents are either home students or great rotators. Our rank lists typically have the well known students listed very high. There’s just no substitute for the personal knowledge of how someone works.

Granted we all know they’re in audition mode for the month so they are probably showing something better than their average effort, but at least we know what’s possible. That said, we probably offer interviews to 10-20% of rotators because many people manage to slip out of audition mode pretty quickly.

Side note: I’ve looked back at rotators we didn’t like and most all of them have gone on to be successful residents in other programs.
Click to expand...
First off, I always enjoy reading you posts and I agree with the above. Residents can slip from audition mode for sure. My experience was more with non rotaters who had good scores, letters and from good schools. They are certainly greater unknowns. Successfully completing the residency is a rather low bar IMO. We have residents who finished, but with great drama and energy drain from our department. With certain residents, a frequent occurrence would be an attending or nurse, red faced stating " Do you know what YOUR resident, said, didnt say, did, didnt do, etc..." My personal experience with these types was more common with residents who didnt rotate with us as students. Just my opinion. It is why I believe having rotated on our service and doing well is my most important metric, but certainly not the only one.
 
Dantrolene FC said:
Away rotations should be against LCME rules. Why should wealthier kids be able to pay money to rotate at a top program to subsequently be ranked higher?

It almost sounds like a politician buying an election.
Click to expand...
most specialties do not require them. Its not really like buying an election, more like being able to intern at a place and getting a job there because people know and like you.
 
Dantrolene FC said:
Away rotations should be against LCME rules. Why should wealthier kids be able to pay money to rotate at a top program to subsequently be ranked higher?

It almost sounds like a politician buying an election.
Click to expand...

Or we can just find a way to standardize costs for auditions as a whole.

To limit PD's simply to a paper application and not have the opportunity to see the applicant truly work with the team would be somewhat sucky.

As an IM applicant who was also tight on cash, I limited myself to rotate at the university hospitals here in the city to save money on room and board. I just made sure to do as well as I could the entire way through. But I can see how it can be frustrating to have to do aways if applying for other specialties.
 
Dantrolene FC said:
Away rotations should be against LCME rules. Why should wealthier kids be able to pay money to rotate at a top program to subsequently be ranked higher?

It almost sounds like a politician buying an election.
Click to expand...
At some newer schools, all rotations are aways. There aren't many months as a 4th yr to audition before interviews begin, only 3 or 4 at my school. Some services wont allow you to do an elective as a 3rd yr. By your screename I would gather you know many anesthesia programs are like that. Could it get expensive? Sure. Life is full of choices and med students have to choose carefully how to advance their career. How do they envision themselves in 10 yrs? Academics? Clinical Med? Where do you want to live?People from Hopkins get paid the same by insurers as people from the Caribbean . If your next 30 yrs are determined by WHERE you do your residency, then you have to do whatever you need to do. I agree with your election analogy. Matching,IMO, is like getting elected. You have to have all the boxes checked. Scores, letters, auditions, research in many cases, networking, etc.
 
Do you think it would be possible to have "Shelf" like exams for the major pre-clinical subject matter?

1) it would help stratify students and residency program directors can focus more on the subject matter more important for that particular specialty
2) NBME would be happy cause they'd make a lot more money
3) I know each schools curriculum is different but just make it required to take the pre-clinical shelves sometime before starting rotations and leave it up to the students to take it when they feel most prepared
 
Dr. Meliodas said:
Do you think it would be possible to have "Shelf" like exams for the major pre-clinical subject matter?

1) it would help stratify students and residency program directors can focus more on the subject matter more important for that particular specialty
2) NBME would be happy cause they'd make a lot more money
3) I know each schools curriculum is different but just make it required to take the pre-clinical shelves sometime before starting rotations and leave it up to the students to take it when they feel most prepared
Click to expand...
they already exist. I took histo, anatomy,micro, pharm, and a handfull of others.
 
Mythoclast said:
I think the question assumes there needs to be a singular metric to judge applicants to begin with. Step 1 wasn't supposed to be a metric in the first place, its a board exam. No employer looks or wants to look at a single metric to decide if they're going to hire someone or not. At least I don't. The long short of it is the AAMC is screwing over residency programs and applicants.

I imagine from the perspective PD's and adcoms, there are a handful of applicants where they're like "Yeah, we want this person.", a few that they definitely don't want, but most fall into the category of "fine, but I don't have a real preference". We suffer from the same problem where I work, however we don't get thousands of applications mostly from people who would go 20 other places before coming here. I think PD's have this problem. So what's a good way to filter everyone? Numbers.

So the problem comes from the way that ERAS is designed. If you want to apply for a job, you can't just apply to every company ever with a click of a button. There are different job posting websites, you have to make cover letters for each company, it's just logistically much harder to do. Now take ERAS, people put in their junk, check boxes, overdraft their bank account and apply to 60 programs. So you've taken a group of highly competitive, highly risk-averse people the ability to send limitless amounts of applications. They're going to assume that if they send more applications than average because they assume that will give them an above average change to match. Now think about that...

The trend you get is that with each generation of pokemon, the average applicant sends about 7 more applications. So where I think we're at now is PD's have to get a PhD in astrophysics to figure out of the 3,000 people who applied to their program, how many of them would rank them highly enough to be worth their time. When in reality they should just be getting a sane amount of applications from the start from people that actually want to go to their program.

Matchmaking systems don't really work when have everyone is ranking everyone. If you're asked to list your preferences of a sufficient number of things, there will come a point where your preferences are no longer rational. The entire system is designed to match you with the programs/applicants you like best, it defeats the purpose if people are applying to and ranking programs they don't really care for.

There has to be strict application limits, fees will not stop people from applying to an ever-increasing number of programs. Statistically most medical students come from very wealthy families, money is no object. Furthermore, if you're already 200K in the hole, was 20k extra debt really gonna stop you? Asking med students to stop sending so many applications is essentially telling them not to worry. It's not going to work, and you look silly for trying.

This will force applicants to be more thoughtful about where they apply, and free programs to actually look beyond a three digit score.
Click to expand...

The problem with the bolded is that it disproportionately harms the people at borders (borderline high to mid-tier person, borderline non-matching people). People who are on the edge for competitive programs would be forced to choose "trying for their dream program" or increasing the likelihood of matching with more safeties.

I actually think hard limits on applications could work, but you can't simply restrict applicants and leave the risk completely up to them. They are already more vulnerable than the programs to which they're applying. If you're going to place hard limits on applications, then you should also provide standardized (i.e. easily identifiable) and very transparent minimum interview requirements. If your program isn't going to interview IMGs or DOs, you should say so. If you're not going to consider anyone with a <215 (or 220, or 230, etc) or below, a fail in a board, less than X number of research experiences for interview, then it should be clear on the website or in the application (ERAS could even have a pop-up that asks if you want to continue with an app whose minimum requirements you don't meet). If those filters were made transparent, then I guarantee applications would drop to manageable levels, especially in combination with a limit on application numbers.

Whenever we talk about these issues, I feel like we always blame the applicants and expect them to change when we create a high risk system where not matching results in a loss of 1 year, loss of income, likely deferment of loans, and possibly even a loss of their entire medical career. Programs can take some responsibility too.

Dr. Meliodas said:
Do you think it would be possible to have "Shelf" like exams for the major pre-clinical subject matter?

1) it would help stratify students and residency program directors can focus more on the subject matter more important for that particular specialty
2) NBME would be happy cause they'd make a lot more money
3) I know each schools curriculum is different but just make it required to take the pre-clinical shelves sometime before starting rotations and leave it up to the students to take it when they feel most prepared
Click to expand...

These already exist. There's one for Physiology, Pharm, Anatomy, etc. I think they're worse than just having Step 1 to be honest.
 
hallowmann said:
The problem with the bolded is that it disproportionately harms the people at borders (borderline high to mid-tier person, borderline non-matching people). People who are on the edge for competitive programs would be forced to choose "trying for their dream program" or increasing the likelihood of matching with more safeties.

I actually think hard limits on applications could work, but you can't simply restrict applicants and leave the risk completely up to them. They are already more vulnerable than the programs to which they're applying. If you're going to place hard limits on applications, then you should also provide standardized (i.e. easily identifiable) and very transparent minimum interview requirements. If your program isn't going to interview IMGs or DOs, you should say so. If you're not going to consider anyone with a <215 (or 220, or 230, etc) or below, a fail in a board, less than X number of research experiences for interview, then it should be clear on the website or in the application (ERAS could even have a pop-up that asks if you want to continue with an app whose minimum requirements you don't meet). If those filters were made transparent, then I guarantee applications would drop to manageable levels, especially in combination with a limit on application numbers.

Whenever we talk about these issues, I feel like we always blame the applicants and expect them to change when we create a high risk system where not matching results in a loss of 1 year, loss of income, likely deferment of loans, and possibly even a loss of their entire medical career. Programs can take some responsibility too.



These already exist. There's one for Physiology, Pharm, Anatomy, etc. I think they're worse than just having Step 1 to be honest.
Click to expand...
The standard errors on the test are much better. 2 points vs 6 for step 1.
 
libertyyne said:
The standard errors on the test are much better. 2 points vs 6 for step 1.
Click to expand...

That may be true, but I wonder what the correlation is between those and specialty board pass rates for example, let alone those and being a good clinician/resident. In that scenario, I think the Steps might actually be a better indicator.
 
libertyyne said:
could you expound on this a little bit. I was under the impression the residents and faculty dont want to send invites to robots either who are constantly in interview mode for the entire month as well.
Click to expand...

Yeah I just mean that you know away rotators are working hard to impress and looking for ways to go above and beyond that they may not do as a resident. Interview mode was not meant as a pejorative - not annoying arse kissing but just consistent hard work.
 
hallowmann said:
The problem with the bolded is that it disproportionately harms the people at borders (borderline high to mid-tier person, borderline non-matching people). People who are on the edge for competitive programs would be forced to choose "trying for their dream program" or increasing the likelihood of matching with more safeties.

I actually think hard limits on applications could work, but you can't simply restrict applicants and leave the risk completely up to them. They are already more vulnerable than the programs to which they're applying. If you're going to place hard limits on applications, then you should also provide standardized (i.e. easily identifiable) and very transparent minimum interview requirements. If your program isn't going to interview IMGs or DOs, you should say so. If you're not going to consider anyone with a <215 (or 220, or 230, etc) or below, a fail in a board, less than X number of research experiences for interview, then it should be clear on the website or in the application (ERAS could even have a pop-up that asks if you want to continue with an app whose minimum requirements you don't meet). If those filters were made transparent, then I guarantee applications would drop to manageable levels, especially in combination with a limit on application numbers.

Whenever we talk about these issues, I feel like we always blame the applicants and expect them to change when we create a high risk system where not matching results in a loss of 1 year, loss of income, likely deferment of loans, and possibly even a loss of their entire medical career. Programs can take some responsibility too.



These already exist. There's one for Physiology, Pharm, Anatomy, etc. I think they're worse than just having Step 1 to be honest.
Click to expand...
They will never release the minimum board score filters because they don’t want transparency in the process. Transparency will show what every suspects - I.e. there are different qualification criteria depending on your educational and gender/race.

The student affairs dean at my school, or whatever his title is, told us they increase the rank order a bit for women and minorities to increase the diversity in the IM program. There’s no way they want the general public to get a hold of that data. Some people, I.e. conservative media, would interpret the data that some doctors are less qualified than others.
 
hallowmann said:
That may be true, but I wonder what the correlation is between those and specialty board pass rates for example, let alone those and being a good clinician/resident. In that scenario, I think the Steps might actually be a better indicator.
Click to expand...

I suspect there is very little correlation between any of these exams and being a good clinician, though I'd be curious to see how step 1 tracks with borad pass rate. Good test takers are good test takers (tautology). Doing well on the exam just means you're organized or smart enough to get all the information in. This in itself is an important thing to know, because in residency you have to assimilate a lot of data.

No one is taking an applicant on the strength of the step 1 score. If you have a hardworking kid who doesn't have that much research from a smaller program that doesn't have much in your specialty. Good grades. Amazing step 1 score... that person still may have a shot at landing at a big program. Without that step 1 score to differentiate that student from the rest, it's tough.

It's like the MCAT or the SAT. The SAT has very little to do with success in college just as the MCAT has very little to do with success in medical school. They're essentially tests of how willing you are to commit yourself to jumping through hoops. But it's an important quality as the in-service, the board exam, fellowship application, buffing your CV for the big academic job... are all exercises in hoop jumping.
 
Angus Avagadro said:
I have sat on resident selection comittees and my number one metric is have we had this person on our service and can I teach them? That's my job. We know if they are hard workers and how they interact with faculty, residents and staff. Some really bright residents have been unteachable. Never rotated with us but had good scores, letters, and good schools. Scores will get you noticed, but auditioning allows us to see the whole package,
Click to expand...
How do you judge if a person is teachable before actually interacting with them over a long period of time? Can you tell from an applicant's ERAS application + interview?
 
You must log in or register to reply here.

Similar threads

S
MD & DO Should I take a practice Step 1 exam just to see where I stand/ What's the best way to study this summer?
Replies
13
Views
956
Med Ed
Med Ed
cerebellar
Step 1 advice!
Replies
0
Views
458
cerebellar
cerebellar
Sram.jpg
Changing From P/F to Letter Grades
Replies
7
Views
1K
14_karat
14_karat
S
M1 Step 1 Advice
Replies
4
Views
466
Goro
Goro
H
What would you do?
Replies
12
Views
2K
operaman
O
Top