Score Creep during MD/PhD training

[Penn-mdphd]

What's some people's opinion on score creep currently (also, @Neuronix)? Will it still be as large a problem from 2018 to 2024 as it's been for the last decade? It's got to plateau at some point right -- hopefully at 229. Hopefully it doesn't rise another 8 points in five years (like 2009-2014).

It's not a primary concern during MD/PhD training obviously, but it does help to be aware of it so that expectations meet reality when it comes time for choosing a specialty and for residency applications.

For reference:
Year Mean Score Standard Deviation
2015 229 20
2014 229 20
2013 228 21
2012 227 22
2011 225 22
2010 222 24
2009 221 24
2008 221 23

A point of note: Standard deviation is going down slightly, indicating the difference between the same two high scores from different years is less than the difference between the same two near average scores from those different years (i.e a 260 in 2014 is more similar to a 260 in 2009 than a 221 in 2014 to a 221 in 2009. i.e. the difference between scores at one standard deviation in 2014 and 2009 is 4 (249-245), which is half the difference between scores at the average in 2014 and 2009 (229-221 = 8).)

Also in other words, if you score very high, you have less to worry about score creep than if you score closer to average for your year. If you score one standard deviation above average, a change of 8 for the average is more realistically a change of 4 for you.

---

Members don't see this ad.

 

[globe199]

Step 1 score creep does adversely affect MD/PhDs, especially if they don't go into the "traditional" IM/Peds/Path/Neuro/Psych specialties that program directors try to push students into. If you score below a residency program's cutoff (whether or not they're in a "traditional" specialty), they'll never even see your PhD! At that point your best bet would be to have phone calls made on your behalf...

The reason score creep happens is because of the availability of better study resources over the years. We can expect the average to go up, but only to a certain point because "raw" test-taking ability plays a larger role at higher scores. Only a small percentage of medical students get above 260 even though we all draw from the same set of resources - hence, the standard deviation of scores is expected to decrease with higher means. Therefore, the higher your score, the less score creep affects you.

I always advise preclinical MSTPs in my program to make the best effort they possibly can on Step 1, just like their regular MD classmates. Doing well on the boards can only help you by keeping more options available to you, regardless of whether or not you're an MD/PhD. Don't listen when someone tells you that "board score doesn't matter".

 

[Neuronix]

What's some people's opinion on score creep currently (also, @Neuronix)?

What's happening is that there are more and more medical students--MDs, DOs, Caribbean, FMGs, etc--and the number of residency spots overall is not keeping up. This means that the competition for those residency positions increases overall, not just in the competitive specialties. One of the things that I wanted to make clear in my paper (published on my blog) is that it's not just the most competitive specialties with inflating step scores. It's all specialties.

Thus, this score creep will likely continue. The number of medical students continues to increase without the same increase in residencies. In general, medicine will become more like law as it becomes more and more necessary to go to a strong medical school and do well there to get a decent (or maybe any) residency.

Where did you obtain the data that you posted? Please provide the source. It's good to keep an eye on this.

There are a lot of naysayers that say these changes don't matter. The numbers may not seem like a lot of growth over six years. But what happens is that a 221 in 2009 goes to a 229 in 2015. For competitive, research oriented programs in any specialty, that growth in scores will change their minimum step score bar from 220 to 230 (everyone likes round numbers). Who wants a "below average" step 1 score resident, despite that the applicant with a 221 might be even more qualified than their residents 10 years ago. Further, MD/PhDs already have a stigma against them that they are not good clinically. Then you have a program director who is typically an MD who is looking for the easiest people to train clinically, not the best researchers. We're not looked at like future grant earning investments anymore because the success rates for getting big grants have been so low for so long. It all means that the MD/PhD doesn't have the same weight it used to, and we have to compete on the same field with MDs. We're not in our own league, as it once was.

I've always worried that the continued stagnant funding and continued growth of MD/PhD programs would cause further rises in the MD/PhD failure to match rate. Fortunately, while the percentage of MD/PhDs going unmatched has fluctuated, the last charting outcomes I checked it didn't continue to increase. I think part of that is that MD/PhD programs have woken up to the fact that their graduates don't get whatever residency they want, run a real risk of not matching, and advise appropriately to save everyone embarrassment. This remains some kind of deep dark hushed secret though, hidden from match lists and public discussion.

 

[gbwillner]

I took the Step I in 2002, the average score was 210. I had a pretty average score (I didn't really prepare for it; I didn't have internet sources like this to tell me how important it would be); I just needed to pass. When I finished med school I was now below average.

Ultimately, it probably did steer my decision making, but I cannot say that the creep really affected me much (still went to a top residency program, fellowship, position).

 

[Penn-mdphd]

Where did you obtain the data that you posted? Please provide the source. It's good to keep an eye on this.

2008 to 2014 is on Wikipedia: https://en.wikipedia.org/wiki/USMLE_Step_1
2013 to 2015 is on http://www.usmle.org/pdfs/transcripts/USMLE_Step_Examination_Score_Interpretation_Guidelines.pdf

Also, just found this on Baylor College of Medicine after googling "Step 1 historical average"
https://www.bcm.edu/education/schools/medical-school/prospective-students/success-indicators
Year | %BCM pass first time test takers | % overall US-Can pass first time | BCM mean first time | Overall US-Can mean first time
2015 100% 95% 246 229
2014 100% 96% 243 229
2013 99% 96% 242 228
2012 99% 95% 240 227
2011 98% 94% 241 224
2010 99% 91% 244 222
2009 100% 94% 242 222
2008 99% 93% 239 221
2007 99% 94% 236 222
2006 99% 93% 237 218
2005 100% 93% 236 217
The numbers are off by 1 in 2009 and 2011 from wiki above

Reminds me of (click to enlarge):

Looks like we're currently at the S-curve maturation Possibly has to do with the explosion of new online study materials in the last years. But that may be plateauing, until a new paradigm comes in.
Graph from Wait But Why's post on artificial intelligence I always enjoy


But yeah, thanks guys!
So I'll take it just do as well as possible regardless.

 

[Fencer]

Some years ago, Neuronix and I submitted a paper to a couple of journals regarding this issue. Reviewers felt that this was easily acquirable data than didn't have any special significance for MD/PhD students. Clearly, we did not explain ourselves well enough...

The last post of his blog shows an essay on this issue:
http://www.neuronix.org/

 

[Neuronix]

Some years ago, Neuronix and I submitted a paper to a couple of journals regarding this issue. Reviewers felt that this was easily acquirable data than didn't have any special significance for MD/PhD students. Clearly, we did not explain ourselves well enough...

The first two journals didn't even review it. The third journal gave me back one review by a reviewer who clearly didn't understand any of the paper.

I got so frustrated with it that I just decided I didn't need another publication. I mean, I had the analysis and it was important to the community, so I self-published it and we can discuss it here.

http://www.neuronix.org/2012/06/effects-of-score-creep-trends-in.html

 

[Shifty B]

This is a real phenomenon and has some effect. How much it will affect you is difficult to predict, because you don't know what the future is going to be like. The best advice possible has already come up though: you have to do your best on Step 1 preemptively, knowing that it's going to affect your future.

I took step 1 in 2003. I didn't even realize that it was a big deal. I was into research and I thought it was pretty much a pass fail test that would let you get a medical license. I did well enough, but not stellar. Maybe about 0.75 SD above the average. Then, a couple of issues came up:

- By the time I matched in 09, the average had probably crept up by 1/2 a SD, so my score was looking a lot closer to average
- I decided to do a relatively competitive specialty (radiology)
- I got married to another physician and couple matched
- I got relatively poor advice that MD/PhDs can match into super-competitive programs easily

While any of those one things might have made it harder, the combination became additive. Our match was a challenge. I ended up at a great program, but we did not necessarily get interviews at all the places we wanted and the stress was higher than expected. The keys are to be realistic and know that it's a competitive process for which you will have to compete with a ton of other MD applicants. This is really hard to foresee 6 years in advance.

 

[quintessences]

Seems like it will be around 240 by the time I get to residency

 

[daolso]

The 2016 Step 1 average was 228 (down 1 point from the year before), with a 21 point standard deviation. It seems we are at a plateau, at least temporarily.

Step 2 continues to rise, however, with another 2 point jump up to 242 (8 points in 6 years).

http://www.usmle.org/pdfs/transcripts/USMLE_Step_Examination_Score_Interpretation_Guidelines.pdf