Impact of board score creep on MD/PhDs?

Discussion in 'Physician Scientists' started by DeadCactus, Sep 11, 2011.

  1. SDN is made possible through member donations, sponsorships, and our volunteers. Learn about SDN's nonprofit mission.
  1. DeadCactus

    DeadCactus SDN Lifetime Donor Lifetime Donor 10+ Year Member

    2,464
    513
    Oct 28, 2006
    I'm curious if those of you doing the MD/PhD program worry about the impact of score creep on your match application. Do you worry that you'll come back out of the lab to find that the scores for your specialty have risen and your previously competitive score is now bellow average?
     
  2. SDN Members don't see this ad. About the ads.
  3. Fencer

    Fencer MD/PhD Director 10+ Year Member

    1,088
    845
    Oct 10, 2007
    TEXAS
    Physician
    PhD
    Faculty
    Unless you go out and spend a decade getting your PhD, this is not relevant. Faculty take time to adapt to what is a good or bad score. Actually, we look back to what we got ourselves as the benchmark. The extra papers and degree will carry you farther in most academic programs than the possible 5 or 10 points of "score creep" on the USMLE part I. The part II is the most important part considered for residency, and is taken when you are back to the last 2 years of medical school at the same time the MD-only peers.

    Sorry, but this was a bellow average question.;)
     
  4. Neuronix

    Neuronix Total nerd SDN Advisor SDN Administrator 10+ Year Member

    12,328
    1,189
    Mar 14, 2002
    the beach
    Physician
    PhD
    Faculty
    I will completely disagree with Fencer. What he said may be true of his specialty. For the specialties I considered and then applied in, his advice is incorrect. I have talked about this at length in the past.

    See:
    http://www.neuronix.org/2011/07/nrmp-puts-out-charting-outcomes-in.html
    http://www.neuronix.org/2011/09/meeting-about-return-this-past-week-i.html

    Step 1, clinical grades, and LORs are the most important factors for competitive specialties. Without them you will not even get an interview. Step 2 is hardly considered. Your actual research prowess is a minor consideration for most programs.

    PS: I wish this weren't the case. Don't shoot the messenger. I just went through this process myself, and I have seen several people from my own school not match at all using bad advice from higher ups in the program. I had what I thought was a fairly strong application with extensive research credentials, but got beat out repeatedly for interviews and then on match day by MD only applicants who had at most a year of research.
     
    Last edited: Sep 13, 2011
  5. M Tobaggen MD

    M Tobaggen MD

    36
    1
    May 16, 2011
    I've heard many people echo the same sentiment as Neuronix.

    When it comes to grades/ step 1, do not aim to get the national average of said speciality. Try to get the highest possible score you can. Even if you're planning not to do a competitive residency, odds are what you want to do will change. Having good scores will never hurt you.

    When it comes to residencies in general, I doubt directors say "I love this candidate but their credentials are way too good."
     
  6. Fencer

    Fencer MD/PhD Director 10+ Year Member

    1,088
    845
    Oct 10, 2007
    TEXAS
    Physician
    PhD
    Faculty
    Please read the OP questions carefully... The questions were not about the importance of USMLE Step I but about slight variations from year to year on the score of Step I (score creep).

    The meaning of a 245 will vary from year to year, but is a top tier score every year because is more than one standard deviation from the mean and median scores.

    In my institution, I have been in the Internal Review board that examines our residencies and fellowships, and have been associate PD for Neurology. There is a large variance for USMLE scores thresholds for particular competitive residencies, which co-varies with the proportion of US Med students applying for those spots. However, most program directors use the LORs and USMLE Step II to discern among the applicants than Step I. Clearly, the more competitive residencies and specialties are going to use Step I even have cut-offs for interviews based upon scores.

    Having said all of this, there are several papers that have been placing questions on the validity of using Step I and/or II as predictors of scores or passing rates of board examinations after residency. Clearly, USMLE scores do not capture the essence of predicting good doctors. :rolleyes:
     
  7. echod

    echod Junior Member 10+ Year Member

    243
    0
    Apr 17, 2006
    I thought Step II scores are best to be taken late unless one did poorly on step I? I thought most LOR were more or less good so how do you differentiate a good letter from another good letter?
     
  8. rad0nc

    rad0nc 2+ Year Member

    186
    1
    Jun 4, 2011
    'Merica
    I am applying to radiation oncology, so maybe this specialty is just totally different. But all three of the program directors I spoke with said that they do not even look at step 2 scores. Step 1 is where it is at for radiation oncology. I've heard similar things for the other competitive specialties.
     
  9. Neuronix

    Neuronix Total nerd SDN Advisor SDN Administrator 10+ Year Member

    12,328
    1,189
    Mar 14, 2002
    the beach
    Physician
    PhD
    Faculty
    For most programs, that is correct. See:
    http://forums.studentdoctor.net/showthread.php?t=829175
    http://forums.studentdoctor.net/showthread.php?t=778027

    This has been discussed at length in many SDN residency forums and the Aunt Minnie radiology forums and I can provide many more links as needed. Also, see:

    Key criteria for selection of radiology residents: results of a national survey., Otero et al, Acad Radiol, 13 (2006):1155-64

    Determinant Criteria for Considering Interview (64/70 programs responding to survey were academic) in order of most to least important:
    1. USMLE scores
    2. Dean's Letter
    3. Class Rank
    4. Recommendations
    5. Honor society membership
    6. Leadership experiences
    7. Research experience
    8. Grade in a specific rotation
    9. Employment experience
    10. Volunteer experience

    The best LORs are LORs from people who got to know you beyond just a clinical rotation and who are big names in the field. Thus, doing research with big names in your future residency field can be extremely helpful. This is one of many reasons to do your PhD in your field if interest. You should be able to obtain LORs from physicians or physician-scientists in your field beyond a letter from a rotation. If you don't do that, spending a few months in fourth year doing research in your field of interest may help your application significantly in this regard. More on this later.

    When I was considering rads and rad onc initially as a second year, the average score for both was 235. When I scored a 240/98 on my step 1, I was very pleased. When I applied, the average score for both was 240. I was told by a home faculty memeber that a 98 two digit score could hurt me at programs ("most strong academic programs want all their residents to be 99th percentile").

    When I found out a 240 step 1 score was merely average and the cutoff score at some programs, I took step 2 early and scored very highly on it. I don't think that really made any difference. But if I had scored the previous average of 235 on step 1, I don't know how many of my 11 out of 45 interviews would have cut me off for having less than a 240 (less than average).

    Why do I care? Big name academic programs where I can get the research training I need want the top residents including the top step scores. Non-academic programs won't interview me at all because they don' have the research I need. This is called the MD/PhD trap and I'd written about it in the article I linked previously.

    So does 5 points of creep in 5 years really make any difference? I'll leave that for the reader to decide.

    I 100% agree. The MCAT does not predict how good of a medical student or physician you will be either. But competition is what it is. Ideally, a successful MD/PhD with many publications would be a valued commodity as long as they were competent in medical school, but this is not grounded in reality for competitive specialties. The 245 you posited as an example is somewhere around the 90th percentile for step 1 scores nationally. If you score at the 51st percentile (say a 221), it is likely you will not match at all in radiology or rad onc. 10 years ago when the average was in the low 230s, this likely would not have been the case (a little longer term score creep there...).

    That said, the arguments I have heard from program faculty are as follows. What the USMLE tests is how good you are at passing board exams. Residency board exam pass rates are very important to residency programs. They want the least chance of a resident failing the boards, as this creates a lot of problems in the program. Similarly, a student's clinical grades predict, if nothing else, how amiable the student will be when dealing with staff, faculty, and scut work. Thus, taking a student who is highest in these arbitrary indicators is a soft predictor of how little of a problem that resident will be.

    Conversely, I have been repeatedly told what a problem MD/PhD residents can be. First, the MD/PhD wants research time that takes them out of clinic. Second, they often don't focus on as much clinical trivia and want to focus on research or other big picture questions. Both of these can make them appear weaker than their co-residents, and puts them at a higher risk of failing their specialty boards. Given that there is no guarantee that an MD/PhD resident will go on to have an academic career, many faculty and PDs do not see the purpose of cutting them any slack clinically or when evaluating them on the MD side for the residency program.

    Further, I have been repeatedly reminded that "not everyone in the field needs to be an academic" and that "we're happy if we take one or two MD/PhDs a year and everyone else canbe clinically focused" by top research programs. If you have a lot of MD/PhD competition, this further hurts you, as you then have to compete with other MD/PhDs if you are in a field that has many of them. Radiology is the 4th most popular specialty for MD/PhDs and there are few strongly academic programs. Radiation oncology has a ton of MD/PhDs. Even when being evaluated against MD/PhDs at the radiology or rad onc programs, I still think AOA and step 1 score are most important, because faculty simply don't know how to evaluate graduate school work. In this regard, doing your research in the field you wish to match in can help you because they get a better sense of your accomplishments and potential. But the benchmark at many programs is still: "you published right?", and thus the research work of a 4th year medical student who published a first author clinical paper or two becomes roughly the same as your 4 years of blood, sweat, and tears.

    The interview and match process was incredibly disheartening for me, because I used to believe strongly in a physician-scientist career for myself. I remain passionate about this topic for the following reason. I was told repeatedly when I stated my program to pace myself and that I would be able to obtain any residecy I wanted as long as I passed step 1 and my coursework. As such, my first and second year performance was average to below average (at my top tier medical school) though I didn't fail anything. It was only when a faculty member at a home program told me I may not match at all coming back from my very successful PhD that I kicked it into high gear.

    And while I honored just about everything when I returned to medical school, that was not enough to push my class rank where it needed to be, and a sky high step 2 doesn't replace an average step 1 score. So when I received no interviews from the region of the country where I wanted to do residency, was shut out by most of the big academic programs, and then began to fear whether I'd match at all as I got the fewest interviews of all the MD-onlies applying from my medical school, I wished I could go back in time and change things.

    Hopefully my warning can serve to help those junior students from making the same mistakes. Even if you don't think you want a competitive specialty or region now, things may change when you meet someone, or you realize rad onc is the coolest specialty in medicine, or that your collaborator happens to be at UCSF and that would be the best place to further your career.
     
  10. vitalamine

    vitalamine 2+ Year Member

    33
    3
    Feb 7, 2011
    I don't mean to distract from the focus of this discussion. Just wondering what the deal is with the 2-digit step 1 score. I scored a 229/99. I assume the 3-digit score is the most important and used for screening. Why does Neuronix's 240 only correspond to a 98 while my 229 gets a 99?
     
  11. M Tobaggen MD

    M Tobaggen MD

    36
    1
    May 16, 2011
    Not positive on the double digit numbers reason for being there but I know residencies now dont even get to see it. Only you can and your school
     
  12. Neuronix

    Neuronix Total nerd SDN Advisor SDN Administrator 10+ Year Member

    12,328
    1,189
    Mar 14, 2002
    the beach
    Physician
    PhD
    Faculty
    See: http://www.usmle.org/general_information/bulletin/2011/score.html

    Nobody outside the USMLE knows exactly how the two digit and three digit scores correlate. Many faculty and PDs I discuss step scores with call it a "percentile", which is false. Some faculty/PDs do use the two digit score when comparing applicants which is aganist the linked score guidelines.
     
  13. QofQuimica

    QofQuimica Seriously, dude, I think you're overreacting.... Lifetime Donor SDN Administrator 10+ Year Member

    This is specialty dependent, and also program dependent. Some programs in my specialty will not rank applicants at all unless they have their Step 2 scores in hand. In general, it seems like the more clinically-oriented specialties with fewer MD/PhDs care more about Step 2 scores, which may be part of the disconnect between Neuro's experiences and Fencer's. (FWIW, my experiences, being in a more clinically-oriented specialty, were more similar to Fencer's than to Neuro's.)

    Like Neuro said, who you have writing the LORs, and how well they know you, is very important. You need to take it upon yourself to get to know senior faculty in your specialty and work with them clinically. In my case, I spent extra time working with the program director at my home program. He got to know me very well and wrote an excellent LOR, to the point that interviewers at other programs made comments like that my school thought I walked on water. When I did my away rotation, I obtained a LOR from that PD as well. He is well known in academic circles, and again, I heard things from interviewers to the effect of, "If Dr. X recommends you, that's all I need to know, because I trust his judgment."

    Ultimately, the best strategy is to have a mentor in your field who is familiar with the app process in that field and can help you maximize your chances of success. I think that being mentored by the PD at my home program is probably the major reason why I was ultimately very successful with my app (35 apps, 29 invites). It should also go without saying that having a good app in and of itself is essential, because the best mentor in the world can only do so much to help you get into your program of choice if you don't have the credentials.
     
  14. solitude

    solitude Senior Member 10+ Year Member

    1,613
    55
    Dec 27, 2004
    perhaps a digression, but can we expect board creep to continue indefinitely? It seems to me that it should level off as students run out of ways that they can maximize their score (UWorld, not going to MS1/MS2 classes, motivation to study more, etc.).
     
  15. Fencer

    Fencer MD/PhD Director 10+ Year Member

    1,088
    845
    Oct 10, 2007
    TEXAS
    Physician
    PhD
    Faculty
    Neuronix demonstrated a great example where the step I score creep made a difference. However, I do not think this is an indefinite increase.
     
  16. Neuronix

    Neuronix Total nerd SDN Advisor SDN Administrator 10+ Year Member

    12,328
    1,189
    Mar 14, 2002
    the beach
    Physician
    PhD
    Faculty
    I agree with Fencer that there can be no indefinite increase as long as the step 1 and step 2 means and standard deviations remain reasonably constant. This is really a supply and demand issue. For competitive specialties, the step 1 and 2 may continue to rise as this is a small subset of select students.

    Though I've crunched the numbers and seen that step 1 and 2 are rising for nearly ALL specialties. I beileve this is a function of an increase in allopathic and osteopathic enrollment. Remember that osteopathic medicine does not have nearly enough residency positions and relies heavily on the allopathic match. In addition, carribean grads and foreign grads are competing for positions. See for example:

    [​IMG]

    This makes the entire match much more competitive. There a number of factors that could change the competition landscape, including increasing residency positions, giving first priority for allopathic residency positions to allopathic graduates, equaliziation of pay across specialties, etc...
     
    Last edited: Sep 15, 2011
  17. Neuronix

    Neuronix Total nerd SDN Advisor SDN Administrator 10+ Year Member

    12,328
    1,189
    Mar 14, 2002
    the beach
    Physician
    PhD
    Faculty
    SurgeonOfLife likes this.
  18. mercaptovizadeh

    mercaptovizadeh ἀλώπηξ 10+ Year Member

    6,662
    561
    Oct 15, 2004
    Clearly, there is creep going on and it does affect MD/PhDs.

    MD/PhD students would do well to remember that - as in ALL things in life - always do your absolute best.

    Try to get the best pre-clinical grades, Step 1 score, PhD publications, Step 2 score, and clinical grades possible. Excelling can never hurt you, and you never know what you'll want to end up doing.

    Even in psychiatry, PM&R, and pediatrics, a 260 is better than a 210.
     
  19. solitude

    solitude Senior Member 10+ Year Member

    1,613
    55
    Dec 27, 2004
    thanks for posting this. Good analysis.

    Ridiculous that the journals didn't feel it was "relevant to residencies". You could have gone PLoS One? But then you would have had to pay a sh*tload out of pocket for publication expenses.
     
  20. bd4727

    bd4727 7+ Year Member

    350
    18
    Jul 17, 2008
    What about a letter instead of a full manuscript?
     
  21. Fencer

    Fencer MD/PhD Director 10+ Year Member

    1,088
    845
    Oct 10, 2007
    TEXAS
    Physician
    PhD
    Faculty
    We tried 3 journals. Two comments reflected the views from reviewers. One indicated that these data were easily available, and the manuscript was not substantially adding to the field. This is an unfair criticism because this particular issue has not been raised. The other criticism was that the reviewer failed to see the connection with MD/PhD students. Perhaps, we could have made a clearer and more concrete connection using a case study. Nevertheless, the main audience is MD/PhD students and their advisors. This thread raises the visibility of the problem. As I posted earlier, I don't think that this is an indefinite trend, but that it mostly reflects at least four factors:
    • a change in the preference for particular residencies (maybe for lifestyle or monetary/debt issues),
    • a greater supply of US medical graduates,
    • a relatively stable number of residency slots, and perhaps
    • better preparation materials for taking USMLE at US medical schools
     
    SurgeonOfLife and Neuronix like this.
  22. miz

    miz 2+ Year Member

    158
    7
    Oct 14, 2011
    There's always PeerJ, I suppose...
     
  23. solitude

    solitude Senior Member 10+ Year Member

    1,613
    55
    Dec 27, 2004

    I'm confused by this. Do you mean you expect this trend to level off?
     
  24. Neuronix

    Neuronix Total nerd SDN Advisor SDN Administrator 10+ Year Member

    12,328
    1,189
    Mar 14, 2002
    the beach
    Physician
    PhD
    Faculty
    I don't measure my career in number of publications. The biggest issue for me is and always has been getting this information to the pre-meds, medical students, and MD/PhDs who would benefit from this data. I regret that I had not just posted this time sensitive data to SDN and/or my blog about 9 months ago.

    The idea of peer review for this data is silly given that anyone has clear access to the source data. If the data is irrelevant because it's too easily obtained, then the NRMP should clearly present their trends over time. If those reading the article don't agree with my discussion topics, that's their choice. But, the data is clear and relevant to medical students, MD/PhDs, and those who advise them.
     
    Last edited: Jun 14, 2012
  25. miz

    miz 2+ Year Member

    158
    7
    Oct 14, 2011
    Thank you for compiling the data, writing the text and sharing it. It's very useful information to know. I'll pass along the link to interested parties.
     
  26. gbwillner

    gbwillner Pastafarian Moderator Emeritus 10+ Year Member

    I guess I will add an additional case study- from my experience.

    When I took the USMLE (2001), the mean score was 210-212. When I applied for residency, the mean score was roughly 220. That means if you are like most applicants and have an AVERAGE score, you are now suddenly a below-average applicant. Sure, if you kill the test and score 245, you will always be OK- but this is a vast minority of applicants. The worst is if you are below average, but within 1 SD of the mean score (let's say you scored a 202 in 2001). You are still average- just a few points worse than the mean. Now you apply in 2008 where the average is nearly a full 20 points higher- you are borderline retarded at this point.

    I agree it is specialty and program dependent- some progams/specialties could care less what you USMLE score is (as long as it is above some arbitrary cut-off, like 200). They want to fill a portion of their resident class with scientists, and will score them differently than other residents (with obvious criteria that are a matter for a separate post). My USMLE score was just average but I had my choice of residency programs in my field (pathology). I was told I was ranked to match in advance by roughly 9/12 programs I applied to. One that did not went out of their way to tell me they do not inform people when they are ranked to match (which could have been a lie, I suppose).
     
  27. erikreinertsen

    erikreinertsen MD/PhD student 5+ Year Member

    96
    1
    Aug 18, 2009
    Atlanta, GA
    So the moral of the story is to ignore the bad advice of "boards don't matter" and ace the step 1? Seems very sensible.
     
  28. Shifty B

    Shifty B 7+ Year Member

    444
    91
    Aug 13, 2008
    Physician
    Yes.

    Board creep is real, but probably cannot go on forever, because the scores will level out. It is a subset of a larger issue, which is that because of the delay to perform a PhD in the middle of medical school you will be judged as part of a different cohort than when you started. So, if step 1 gets more emphasized, USMLEworld becomes more popular, and scores start to drift up during those 4 years, that could hurt you. If your medical school becomes more competitive (due to changes in the economy, changes in US News rankings, whatever), then you may find yourself in the middle of your class when you used to be in the top 15%.

    Alternatively, you could find yourself interested in a specialty that is very trendy now but wasn't so trendy 4-5 years ago (ER, anesthesia). You may have been a super applicant when your medical school class finished only to find yourself average now. With the to-and-fro fluctuations in applicants for each specialty, there are more or less competitive years in every specialty.

    Medical schools keep increasing in size but residencies stay about the same as well, but that's a separate issue.

    The only defense you have against that is to ace your clinical grades, and steps 1/2. Since clinical grades can be quite arbitrary, and you are at a disadvantage of having 4 years off and coming back, the most predictable thing that you have control over is step 1. Study more, and you will do better.
     
  29. 24thGrade

    24thGrade 2+ Year Member

    92
    0
    Jul 24, 2010
    Also, this is anecdotal, but when I posted awhile back asking about people applying to PSTPs or whatever you want to call them (IM fast track +/- guaranteed fellowship) the only person who ended up replying felt like during the interview process they were surrounded by all-stars with 240+ board scores. So even if IM and research is your goal Step 1 might still really matter a lot.
     

Share This Page