NRMP data and predicting the odds of matching

[Blunt Dissection]

NRMP recently added an interactive form of their charting outcomes which really makes it easy to see a number of factors and their influence on matching to certain specialties. If this interactive form is complete in its data set over the years, I think it provides some really great insight on understanding the matching process a little better.

Using plastic surgery as an example, it is common knowledge that plastics is an extremely difficult field to match into regardless of being an MD or DO student. People are often fixated on the yearly specialty Step 1 average, which over the years has climbed to 250 for plastics. Yet last year, there was apparently an individual with a <200 Step 1 score that matched. I'm in no way saying that if you scored <200 your odds of matching plastics is good, but it does show the range of scores even in the most competitive of specialties and how a significant amount of factors beyond Step 1 can play into matching. Looking at the DO applicants that applied for plastics, it shows that since 2014, there have only been 12 applicants and of those 12 applicants, 4 have matched. This really shines some light on the number of applicants to each specialty and makes statements like "Plastics is harder for DOs" difficult to defend when there have only been 12 applicants that have applied in the past 4 years.

It makes me curious then if SDN users place more emphasis on the Step 1 and on MD versus DO than is necessary for actual residency placement process. Thoughts?


Tableau Public

---

Members don't see this ad.

 

[watermelon master]

Looking at the DO applicants that applied for plastics, it shows that since 2014, there have only been 12 applicants and of those 12 applicants, 4 have matched. This really shines some light on the number of applicants to each specialty and makes statements like "Plastics is harder for DOs" difficult to defend when there have only been 12 applicants that have applied in the past 4 years.

Or could it be that there is an extreme self-selection bias in that many DOs don't even try to apply since they know that it is out of reach?

 

[Blunt Dissection]

Or could it be that there is an extreme self-selection bias in that many DOs don't even try to apply since they know that it is out of reach?

I think this probably has some role in the matter as does the tendency of DO students to trend towards primary care specialties to begin with. Its curious because when the merger is complete in 2020, will DOs self-exclude themselves from specialties because of this fear?

 

[bananafish94]

Extreme outliers like that are not representative and likely had some kind of ridiculous connection that they knew would give them an in (e.g., their relative is a PD for a program's plastic surgery department or something like that). From my experience SDN posters seem to be more interested in the specialties for which there is the strongest correlation of matching to Step 1. If you look the data there's only a handful of specialties where there's a strong correlation of matching as you increase above a Step 1 score of 230 or so, like orthopedics and whatnot. For most specialties it seems to me that there is significantly diminished returns as you get into the 230s and beyond. So it really depends on what specialty you want to do and how important certain aspects of a program are to you (academic, reputation, etc.)

 

[DoctorPlzzz]

Or could it be that there is an extreme self-selection bias in that many DOs don't even try to apply since they know that it is out of reach?

I think there is some truth to this.

Also, I believe it is exposure during medical school as well. I know that most DO schools have clinical years at different locations that are not usually part of large academic centers. This could limit the availability of a plastics rotations, of finding a plastic surgery mentor/advisor, and of getting LORs from plastics surgeons. I would imagine that getting plastics away rotations would also be harder in this instance as they are competitive. (using plastics as the example)

 

[Goro]

NRMP recently added an interactive form of their charting outcomes which really makes it easy to see a number of factors and their influence on matching to certain specialties. If this interactive form is complete in its data set over the years, I think it provides some really great insight on understanding the matching process a little better.

Using plastic surgery as an example, it is common knowledge that plastics is an extremely difficult field to match into regardless of being an MD or DO student. People are often fixated on the yearly specialty Step 1 average, which over the years has climbed to 250 for plastics. Yet last year, there was apparently an individual with a <200 Step 1 score that matched. I'm in no way saying that if you scored <200 your odds of matching plastics is good, but it does show the range of scores even in the most competitive of specialties and how a significant amount of factors beyond Step 1 can play into matching. Looking at the DO applicants that applied for plastics, it shows that since 2014, there have only been 12 applicants and of those 12 applicants, 4 have matched. This really shines some light on the number of applicants to each specialty and makes statements like "Plastics is harder for DOs" difficult to defend when there have only been 12 applicants that have applied in the past 4 years.

It makes me curious then if SDN users place more emphasis on the Step 1 and on MD versus DO than is necessary for actual residency placement process. Thoughts?

The way some SDNers carry on about it, you'd think that their medical education begins and ends with Step I.

There's a lot of self-selection all over the specialty spectrum for both MD and DO (Using MSAR, look at how few people go into Gen Surg, for example, as a % of any individual school's graduates). A lot of my students from Day 1 gravitate to Primary Care by choice.

I suspect that as with the MCAT, while a Step I score can strongly influence your getting an II, getting accepted into a program is 100% on you.

 

[Dr. Doctor MD]

Extreme outliers like that are not representative and likely had some kind of ridiculous connection that they knew would give them an in (e.g., their relative is a PD for a program's plastic surgery department or something like that). From my experience SDN posters seem to be more interested in the specialties for which there is the strongest correlation of matching to Step 1. If you look the data there's only a handful of specialties where there's a strong correlation of matching as you increase above a Step 1 score of 230 or so, like orthopedics and whatnot. For most specialties it seems to me that there is significantly diminished returns as you get into the 230s and beyond. So it really depends on what specialty you want to do and how important certain aspects of a program are to you (academic, reputation, etc.)

Excellent Answer ^^^

The charting outcomes data is severely lacking. It doesn't include the 100s or 1000s of applicants each year who get 0 interviews and don't even get to make a match list in plastics or ortho or IR. They aren't even in the denominator. I once knew a guy applying to dermatology with 238 and he didn't even get a home program interview. Literally 0 dermatology interviews at all. He ended up soaping into IM. He won't show up in the data.

Most DO plastic surgery applicants will get 0 interviews and maybe 1 if they're incredibly lucky. Among the few that end up even applying only 33% 4/12 match and that's after the self selection. The DOs that match only get like 1-4 interviews max.

It's all about connections. If you saw another 100 DOs apply plastic surgery, MAYBE the match rate would go from 4 total to 7 total, and the percent would go from 33% to 10%. The point I'm making is that you can't draw broad conclusions from the data.

Charting Outcomes is great for IM, Anesthesia,Family and all the other massive regular specialties. It's rubbish for the highly selective fields like plastics neurosurgery or dermatology. It ignores all the important factors that don't show up and excludes the hoards of unqualified applicants who get 0 interviews. Hell, for plastics people with a <200 have a 100% match rate... Does anyone actually think that is real data?

I promise you that anyone applying to Plastics, IR, etc with sub 240 doesn't just have balls, they have some killer connections, research, experiences that make them highly desirable, but you'll never see that in Charting Outcomes.

 

[Blunt Dissection]

The way some SDNers carry on about it, you'd think that their medical education begins and ends with Step I.

There's a lot of self-selection all over the specialty spectrum for both MD and DO (Using MSAR, look at how few people go into Gen Surg, for example, as a % of any individual school's graduates). A lot of my students from Day 1 gravitate to Primary Care by choice.

I suspect that as with the MCAT, while a Step I score can strongly influence your getting an II, getting accepted into a program is 100% on you.

Excellent Answer ^^^

The charting outcomes data is severely lacking. It doesn't include the 100s or 1000s of applicants each year who get 0 interviews and don't even get to make a match list in plastics or ortho or IR. They aren't even in the denominator. I once knew a guy applying to dermatology with 238 and he didn't even get a home program interview. Literally 0 dermatology interviews at all. He ended up soaping into IM. He won't show up in the data.

Most DO plastic surgery applicants will get 0 interviews and maybe 1 if they're incredibly lucky. Among the few that end up even applying only 33% 4/12 match and that's after the self selection. The DOs that match only get like 1-4 interviews max.

It's all about connections. If you saw another 100 DOs apply plastic surgery, MAYBE the match rate would go from 4 total to 7 total, and the percent would go from 33% to 10%. The point I'm making is that you can't draw broad conclusions from the data.

Charting Outcomes is great for IM, Anesthesia,Family and all the other massive regular specialties. It's rubbish for the highly selective fields like plastics neurosurgery or dermatology. It ignores all the important factors that don't show up and excludes the hoards of unqualified applicants who get 0 interviews. Hell, for plastics people with a <200 have a 100% match rate... Does anyone actually think that is real data?

I promise you that anyone applying to Plastics, IR, etc with sub 240 doesn't just have balls, they have some killer connections, research, experiences that make them highly desirable, but you'll never see that in Charting Outcomes.

This is really interesting because while there's countless discussions on board scores on the forums, there's not a ton of discussion on developing the "other" part of the application. Outside of the rare "PD is my relative/family friend/friend of friend" situation, what connections would be legitimately useful? If research directly equals # of publications as suggested by the nrmp data, would more pubs in less significant areas be more "useful" than less pubs with greater significance? What "skills" could medical students really develop that would make a program director prefer candidates that would otherwise be equal on paper? The NRMP program director surveys are useful at getting a peek at some of this, but it also opens the doors for more questions such as for program directors that have said they would not consider a DO student or would not consider a Step 1 score below XXX. Does this mean that these programs have an electronic filtering system similar to how a low MCAT would get you automatically filtered out of certain medical schools?

If applicant A had a Step 1 score of 210, Step 2 score >250, >20 publications, 3rd and 4th year honors, would they get screened out because of that 210?
If applicant B had a Step 1 score of >250, Step 2 >250, 0 publications, average clinicals, would they be more valued than applicant A?

In short - does an auto-screen still exist at this level of education?

 

[Dr. Doctor MD]

This is really interesting because while there's countless discussions on board scores on the forums, there's not a ton of discussion on developing the "other" part of the application. Outside of the rare "PD is my relative/family friend/friend of friend" situation, what connections would be legitimately useful? If research directly equals # of publications as suggested by the nrmp data, would more pubs in less significant areas be more "useful" than less pubs with greater significance? What "skills" could medical students really develop that would make a program director prefer candidates that would otherwise be equal on paper? The NRMP program director surveys are useful at getting a peek at some of this, but it also opens the doors for more questions such as for program directors that have said they would not consider a DO student or would not consider a Step 1 score below XXX. Does this mean that these programs have an electronic filtering system similar to how a low MCAT would get you automatically filtered out of certain medical schools?

If applicant A had a Step 1 score of 210, Step 2 score >250, >20 publications, 3rd and 4th year honors, would they get screened out because of that 210?
If applicant B had a Step 1 score of >250, Step 2 >250, 0 publications, average clinicals, would they be more valued than applicant A?

In short - does an auto-screen still exist at this level of education?

Let me give you a short list of some "other" qualifications
-Applicant has PhD with significant research exposure
-Applicant has and continues to publish actual high impact research in high impact journals
-Applicant has letters from some of the titans of a field both research/clinical saying they are the best student they've ever seen.
-Applicant has family member on faculty
-Applicant knows home department well and they love him/her
-Applicant blows their away rotations out of the water, as in top 5% of rotators
-Applicant was part of a project that advanced the field of interest
-Applicant is an IMG who already did a full residency in field of interest.
-Applicant has all other boxes checked: research, AOA, Honors, Class Rank, Letters, Great Personality.

Any one of these and many more could lead to PDs saying, forget step 1, I really really really want this person at my program and don't care what his/her Step 1 score is.

 

[Blunt Dissection]

Let me give you a short list of some "other" qualifications
-Applicant has PhD with significant research exposure
-Applicant has and continues to publish actual high impact research in high impact journals
-Applicant has letters from some of the titans of a field both research/clinical saying they are the best student they've ever seen.
-Applicant has family member on faculty
-Applicant knows home department well and they love him/her
-Applicant blows their away rotations out of the water, as in top 5% of rotators
-Applicant was part of a project that advanced the field of interest
-Applicant is an IMG who already did a full residency in field of interest.
-Applicant has all other boxes checked: research, AOA, Honors, Class Rank, Letters, Great Personality.

Any one of these and many more could lead to PDs saying, forget step 1, I really really really want this person at my program and don't care what his/her Step 1 score is.

Which sounds to me like what you're saying is the auto-screen really doesn't exist anymore at this level since a lot of those factors (at least those not involving some form of direct connection to the applicant) would mean the PD would have to spend some time looking at the applicants complete file. In this case, I can actually see that being done for smaller, but more selective fields like plastics/derm since the volume of applications is significantly less than fields like anesthesia, emergency, etc.

 

[Goro]

Let me give you a short list of some "other" qualifications
-Applicant has PhD with significant research exposure
-Applicant has and continues to publish actual high impact research in high impact journals
-Applicant has letters from some of the titans of a field both research/clinical saying they are the best student they've ever seen.
-Applicant has family member on faculty
-Applicant knows home department well and they love him/her
-Applicant blows their away rotations out of the water, as in top 5% of rotators
-Applicant was part of a project that advanced the field of interest
-Applicant is an IMG who already did a full residency in field of interest.
-Applicant has all other boxes checked: research, AOA, Honors, Class Rank, Letters, Great Personality.

Any one of these and many more could lead to PDs saying, forget step 1, I really really really want this person at my program and don't care what his/her Step 1 score is.

Connections always count for something.

The Program Director's survey lets you know what PDs want in addition to Board scores.

 

[AnatomyGrey12]

he charting outcomes data is severely lacking. It doesn't include the 100s or 1000s of applicants each year who get 0 interviews and don't even get to make a match list in plastics or ortho or IR. They aren't even in the denominator. I once knew a guy applying to dermatology with 238 and he didn't even get a home program interview. Literally 0 dermatology interviews at all. He ended up soaping into IM. He won't show up in the data.

I agree with the principle but we do have access to the ERAS data I believe, which is submitted applications. There aren't "100s or 1000s" of applicants applying to competitive specialties that aren't getting interviews.

 

[NickNaylor]

I've found that discussions of the NRMP data don't correctly interpret what is being written. For example, mean step 1 scores are just that - means. The NRMP fortunately breaks the data down more granularly than that, but even in "competitive" fields people that might be written off as completely hopeless based on step 1 scores, grades, etc. still match every year.

I think the more interesting pieces of data is surveys of program directors asking them to rank the relative importance of various components of the application by specialty. This is often where this is great different among the fields. In psychiatry, for example, factors that matter for giving an interview vs. ranking high on the match list are quite different when you look at the top 3-5 factors.