"In my experience, USMLE performance clearly predicts residency performance -- those with low scores do not do well." -aPD
There was a study done in 2007 that showed a significant correlation between Step 1 performance and passage of the Pediatric board examination. The authors speculated that by knowing that poor Step 1 performance correlates with failure of the Pediatric board then these residents could have intervention early in residency to help them study better.
In the end, clinical evaluation anywhere is subjective, and sure I have seen students who could smooze their way to a passing grade even while having failed important examinations. The same thing happens in residency where soemtimes residents who
act very competent are not. There is a reason for having inservice exams as attendings in general have pretty subjective evaluations.
In Ob/Gyn those with a board score above 200 were seven times as likely to obtain board certification than those with a score below 200.
So, I think that USMLE is good in that it is an objective measure of clinical knwoledge and clinical management.
I think that some residents can eventually and easily outperform their Step 1/2 score because time is on your side. If you read everyday and read about your patients then you will catch up with your fellow residents who by and large studied more efficiently than you in medical school. This is an uphill battle, but such motivated students are doubtlessly spotted by residency program directors who want someone who will study in residency and plough through the material.
Most excellent internal medicine programs want applicants with relatively high Step 1 scores, i.e. above 230, and nowadays having a high Step 2 CK helps as well.
Doing a study in such a population of residents at MGH in terms of correlation between board score and performance in residency and comparing the results to other institutions is flawed. A mediocre or lukewarm good resident at MGH likely would be a superstart at Podunk Internal Medicine residency.
On the whole residents at big academic centers with high Step 1's really know what they are doing, but some even there some stand out. So if you have someone at Podunk Internal Medicine program with an average board score who is rated as "excellent" by faculty versus someone who has 250 at MGH who is just "average" then it would look like there is no correlation between board scores.
I have seen interns who didn't make it in residency and they had knowledge deficits and poor clinical solving skills, i.e. poor Step 2 CK scores as well as no motivation. There is a study of students who failed Step 1, I will try to find the info. In the end, a PD who asks about a student who failed Step 1 is really trying to see if the student can rise above it as a resident. Step 1 is not rocket science or neurosurgery, if you fail this exam you probably weren't motivated to study medicine, or had a pretty big extenuating circumstance.
In the end I think there are cohorts:
1. Failed Step 1 (or 2). This student needs to re-motivate themselves to learn medicine, and will be judged by Step 2 CK score and clerkship grades and can redeem themselves, but motivation judged in residency interview.
2. Step 1 (or 2) - 187-200. Just passed, obviously below average. Need to also look at clerkship grades and letters to gauge level of committement and to see if the residency program can help such a student bloom into an excellent doctor. I think interviewers try to see if the student understand what went wrong and how to fix it. More high-pressued residencies like surgery, top-notch IM might not benefit such a student as much as others, but motivation for field still a factor. Some residencies will filter out such applicants as the program moves at a fast pace and such a resident may need extra time to figure out what they don't know.
3. Step 1 (or 2) 200-230 This student had some level of passion for medicine, has an adequate knowledge base and could become excellent in residency if he/she learns their field well and is diligent when taking care of patients. Residency program directors like to see how motivated such a person is for say pediatrics. If applicant says that it took them a while to get down level of studying needed for med school, but love their peds core and working with kids then they would be expected to do well.
4. Step 1 (or 2) 230-250 Something really clicked well for this student so he/she could really learn some medicine. Major question is "fit" of the resident with the program, i.e. can they reproduce this result in the environment of your residency program.
5. Step 1 (or 2) > 250. These students by a large number self-select to competitive surgical residencies, dermatology, and top-notch medicine programs. Generally will grows roots and bloom wherever they are planted, or so is assumed. Also want to see if has similar personality to residents.
I think if a high Step 1/2 CK score reflects a lifelong commitment to learn medicine then it is significant especially with medicine becoming more complicated by the day and requiring more reading to stay competent.
If most of a program director's residents are in Group 2 or 3, then there might not be much correlation between residency performance and usmle, although a lot of studies show a correlation between one or the other of the steps and inservice training examinations.