Statistics question on Charting Outcomes

[Abaris]

---

Members do not see this ad.

I noticed that on Charting Outcomes in the Match, it looks like there's a significant advantage in some specialties for IMGs who have *fewer* than 16 ranks.

For example, In Internal Medicine (see page 98), about 85% of IMGs with between 11 and 15 ranks were matched, but only about 67% of those with 16 or more were.

In Psychiatry (see page 240), about 90% of IMGs with between 11 and 15 ranks were matched, but only about 54% of candidates with 16 or more were.

This pattern repeats across most of the larger specialties.

Obviously it would be a bad idea to actively take ranks off my list, but it would make me feel a lot less anxious if someone could come up with a good explanation of this. Anyone have any ideas?

 

[shan564]

I've thought about this before. I've come up with a few possible explanations, but I'm not sure how they analyze their data, so I don't know if this is accurate.

1. People with 16+ ranks might have ended up taking a lot of prematches and withdrawing from the Match.
2. People with 16+ ranks might be interviewing in multiple specialties... so while 67% of them might have matched into IM, the other 33% might have matched elsewhere.
3. Some people might have ranked programs where they didn't actually interview. As you can imagine, these people probably had long rank lists and low match rates.

 

[DrSnips]

I noticed that on Charting Outcomes in the Match, it looks like there's a significant advantage in some specialties for IMGs who have *fewer* than 16 ranks.

For example, In Internal Medicine (see page 98), about 85% of IMGs with between 11 and 15 ranks were matched, but only about 67% of those with 16 or more were.

In Psychiatry (see page 240), about 90% of IMGs with between 11 and 15 ranks were matched, but only about 54% of candidates with 16 or more were.

This pattern repeats across most of the larger specialties.

Obviously it would be a bad idea to actively take ranks off my list, but it would make me feel a lot less anxious if someone could come up with a good explanation of this. Anyone have any ideas?

Correlation is not causation. Examine the match algorithm and you will see it is logically impossible that ranking more places can reduce your chance of matching. Perhaps people who were more competitive did not feel the need to interview at or rank as many places, thus their shorter rank lists are a sign of their success, not the cause of it.

 

[p30doc]

I've thought about this before. I've come up with a few possible explanations, but I'm not sure how they analyze their data, so I don't know if this is accurate.

1. People with 16+ ranks might have ended up taking a lot of prematches and withdrawing from the Match.
2. People with 16+ ranks might be interviewing in multiple specialties... so while 67% of them might have matched into IM, the other 33% might have matched elsewhere.
3. Some people might have ranked programs where they didn't actually interview. As you can imagine, these people probably had long rank lists and low match rates.

I've attributed this phenomenon to #3, no clue if that is the actual reason or not.

 

[K31]

I've thought about this before. I've come up with a few possible explanations, but I'm not sure how they analyze their data, so I don't know if this is accurate.

1. People with 16+ ranks might have ended up taking a lot of prematches and withdrawing from the Match.
2. People with 16+ ranks might be interviewing in multiple specialties... so while 67% of them might have matched into IM, the other 33% might have matched elsewhere.
3. Some people might have ranked programs where they didn't actually interview. As you can imagine, these people probably had long rank lists and low match rates.

1) These people never actually go through the Match (when you accept a prematch you must withdraw from the Match), so they wouldn't be included in the statistics.

2) The number of ranks vs chance of matching only takes into account contiguous ranks in the preferred specialty (the specialty ranked in the #1 position). Say an applicant applies to RadOnc with FM as a backup, gets 2 RadOnc interviews and 16 FM interviews, and ranks them all with the RadOnc programs first then FM. If the applicant matches at either of the RadOnc programs, they'll be included in the RadOnc stats next year. If they match to any FM program, they'll still be included in the RadOnc statistics as an unmatched applicant--not in the FM statistics.

3) I think it must be this--I don't get the logic of ranking programs where you didn't interview (hoping they'll enter your name on their list by accident?) but apparently it does happen.

 

[Ronin786]

1) These people never actually go through the Match (when you accept a prematch you must withdraw from the Match), so they wouldn't be included in the statistics.

2) The number of ranks vs chance of matching only takes into account contiguous ranks in the preferred specialty (the specialty ranked in the #1 position). Say an applicant applies to RadOnc with FM as a backup, gets 2 RadOnc interviews and 16 FM interviews, and ranks them all with the RadOnc programs first then FM. If the applicant matches at either of the RadOnc programs, they'll be included in the RadOnc stats next year. If they match to any FM program, they'll still be included in the RadOnc statistics as an unmatched applicant--not in the FM statistics.

3) I think it must be this--I don't get the logic of ranking programs where you didn't interview (hoping they'll enter your name on their list by accident?) but apparently it does happen.

It's extremely possible that number 3 is the correct answer. There is a good number of IMGs who really don't pay attention to how the match works. Some may consider a waiting list email as cause to rank.

 

[Residentcool]

It's extremely possible that number 3 is the correct answer. There is a good number of IMGs who really don't pay attention to how the match works. Some may consider a waiting list email as cause to rank.

There are some programs that do rank people who have rotated with them even though they are not required to interview. Not sure if this has any relevance.
But I think what someone suggested above is correct regarding the various specialty applying.

 

[sunset823]

I was wondering about this too, because even for US seniors there is a drop for 16+ ranks. I think it is a combo of #2 and #3. Someone mentioned above that a person applying multiple specialties would only be ranked in their preferred specialty, but this is not true. I have several friends applying to 2 specialties and 'mixing' their rank list, it's not like all of one specialty is at the top and all the rest are the other ones. Therefore, if someone is dual applying, say peds and fm, and has a mix, then if they match in peds, they will be 'unmatched' in FM and vice versa, and would show up as unmatched in the Charting Outcomes.

I think this accounts for the vast majority of the unmatched US seniors, cuz we know the rules of not ranking programs we didn't interview at. For independent applicants, #3 might come into play too.

Either way, it doesn't matter, more interviews is always better. Now I'm sort of regretting cancelling 1/3 of my interview invites, even though I have a rank list of 15 (14 programs with 2 tracks on one of them).

 

[K31]

I was wondering about this too, because even for US seniors there is a drop for 16+ ranks. I think it is a combo of #2 and #3. Someone mentioned above that a person applying multiple specialties would only be ranked in their preferred specialty, but this is not true. I have several friends applying to 2 specialties and 'mixing' their rank list, it's not like all of one specialty is at the top and all the rest are the other ones. Therefore, if someone is dual applying, say peds and fm, and has a mix, then if they match in peds, they will be 'unmatched' in FM and vice versa, and would show up as unmatched in the Charting Outcomes.

Read the intro to Charting Outcomes--how the data are compiled all depends on whatprogram you rank first. They count "match success" only as matching to the specialty of the program you rank #1 (the "specialty of choice").

If, as in your example, someone ranked an FM program number 1 and a peds program #2 and matched at the peds program, they'd go down as unmatched in FM (with one contiguous rank) in the statistics, and would not be included at all in the peds statistics. If they had another FM program ranked #3 and matched there, they would go down as matched in FM (again with one contiguous rank, I presume) and again would not be included in the peds statistics.

 

[sunset823]

I stand corrected. Then I have no idea why 27 US seniors with 16+ contiguous ranks in IM were unmatched, if someone can shed light on that, that would be appreciated.

 

[shan564]

I stand corrected. Then I have no idea why 27 US seniors with 16+ contiguous ranks in IM were unmatched, if someone can shed light on that, that would be appreciated.

Probably the same reason that we suggested earlier... people might be ranking programs where they didn't interview, just for the heck of it. Even though they'd never actually match at those places, it doesn't hurt to rank them. I'll bet none of those people ranked 20+ places, since that's when you have to pay extra...

 

[Raryn]

There's always option 4. Some of the people with 16+ interviews only went on 16+ interviews because of some red flag on their app. i.e. the person with multiple step failures who applied to 100+ programs and went on every interview they could get their hands on, because (s)he knew that none of the programs would rank them highly. Or the person who's really really awkward in person and went on that many interviews to try to hedge their bets. etc.

I always assumed the red flag theory was a pretty large portion of the drop off.

 

[Jessipeds]

does a program ever rank someone that wasn't offered an interview? Like if you were on their waitlist?

 

[kindasorta]

does a program ever rank someone that wasn't offered an interview? Like if you were on their waitlist?

I can't fathom that happening. That said, throw 'em on your list. Can't hurt.

 

[Abaris]

There's always option 4. Some of the people with 16+ interviews only went on 16+ interviews because of some red flag on their app. i.e. the person with multiple step failures who applied to 100+ programs and went on every interview they could get their hands on, because (s)he knew that none of the programs would rank them highly. Or the person who's really really awkward in person and went on that many interviews to try to hedge their bets. etc.

I thought about that, but my experience as an international student with what I consider a decent application (passed all steps, no obvious red flags) is that you apply to as many places as you can anyway and take whatever interviews you can get.

This seems to be borne out by the data - even though international students are less likely to match than US students, they also rank fewer places. The average US grad who matches to IM ranks 10 places; the average IMG who matches to IM ranks 5. Why would an IMG stop at 5 interviews if they could possibly get 10? I know there are time and money issues, but why would these deter good candidates more than bad ones? Also, how would this IMG with all these red flags even get 16 interviews?

 

[Ronin786]

I thought about that, but my experience as an international student with what I consider a decent application (passed all steps, no obvious red flags) is that you apply to as many places as you can anyway and take whatever interviews you can get.

This seems to be borne out by the data - even though international students are less likely to match than US students, they also rank fewer places. The average US grad who matches to IM ranks 10 places; the average IMG who matches to IM ranks 5. Why would an IMG stop at 5 interviews if they could possibly get 10? I know there are time and money issues, but why would these deter good candidates more than bad ones? Also, how would this IMG with all these red flags even get 16 interviews?

Exactly. If an IMG is getting 16 interviews then he probably doesnt have any glaring red flags.

On the other hand, it is possible that you have superstar IMGs with a lot of experience and good Step scores, but they're unable to interview well and/or have language troubles. They might go to all those interviews, but bomb them all due to personality or language barriers. That still wouldnt explain the large number of unmatched people with contiguous ranks, but that and the ranking other programs theory might help explain it.

 

[Ronin786]

I thought of another scenario for this. If these specialties have categorical and advanced positions, then you would only need half the number of interviews.

So if someone received 10 interviews, but ranked each categorical and advanced position, they might count as 20 contiguous ranks.

 

[Stellar Clouds]

I thought of another scenario for this. If these specialties have categorical and advanced positions, then you would only need half the number of interviews.

So if someone received 10 interviews, but ranked each categorical and advanced position, they might count as 20 contiguous ranks.

+1

I think you are on to something here. Think about gen surg or IM--desperate people might be ranking both prelim and categorical. And then anesthesia for example, I went on 11 interviews but I have 18 contiguous ranks (categorial and advanced).

 

[Target2]

Has anyone found any data on SOAP outcomes from last year? I searched all through the NRMP website and there's nothing on SOAP results. I'm specifically looking for statistics on how many people applied through the SOAP, how many candidates programs interviewed and how many applicants got positions through the SOAP.

 

[Abaris]

I finally found the Applicant Survey, which seems to confirm that up to 10% of international students are ranking programs they didn't interview at.

But this one is also confusing. It lists matched and unmatched applicants, for example, as having pretty much the same number of interviews and programs ranked, which contradicts Charting Outcomes (they use median instead of mean, but it's easy to get median from Charting Outcomes and that doesn't agree either). Maybe response bias to the survey or something?

 

[GoSpursGo]

does a program ever rank someone that wasn't offered an interview? Like if you were on their waitlist?

No.