USCAP abstract - boards failure correlates with USMLE score

[lipomas]

---

Members do not see this ad.

There is an abstract in the USCAP series published in modern path (# 551) which is from UPMC and states a correlation between USMLE score (2 digit score) and ABP boards pass rate. Takes a lot of balls to actually publish your own institution's boards pass rate and what USMLE scores your residents have. Of course, in my impression UPMC residents have gotten a lot better since 2001 when their study begins.

They have a chart there which states their first time pass rate is 92.7% (national average 82.8%) for AP, 78.9% for CP (average 73.8%). I wonder if that is comparable to other large institutions? I know a lot of them tell you on interviews that "everyone passes the boards" but I suspect they don't always mean that everyone passes on the first try.

No one who had a USMLE score >90 failed. <80 had an overall failure rate of 42.9%, which was higher among US grads (all 3 failed) as compared to IMGs (3 of 11 failed). Scores 80-89 were in the middle, but still higher failure rates for US grads. This is part of the reason why USMLEs are so important for residency matching. The correlation is not surprising - good test takers tend to remain good test takers even when the test gets more difficult. It's also more rationale to the skepticism that is given to US grad applicants who score under 80.

I think it's great that they submitted this study - good data!

 

[gbwillner]

how does the 2-digit score compare to the 3-digit score? I only remember averages and scores as the 3-digit type.

My first response to this information is ...duh. The real question is whether it correlates to actual clinical performance (faculty evaluations, numbers of lawsuits against, earnings, etc.)

 

[mrp]

Vaguely related--did you see the paper in Modern Pathology about the outcome of abstracts presented at USCAP? Only 30-40% are ever published.

 

[2121115]

No surprise there, but it is nice to see it published. It is no shocker that if you take a group of medical students who made 240+ on Step I (MGH, BWH, Hopkins, Duke, Wash U, UCSF, Stanford, etc, etc, etc) and give them a book and some time to study that they will pass the pathology boards. That is the definition of "low risk". I would also bet this correlation is more significant than other measures, like RISE scores.

Also, a two digit 90 on Step I isn't all that high. I think that is right around the average.

 

[2121115]

I just looked at the abstract.

FAIL.

They are reporting the 2 digit score as percentile. The 2 digit score is NOT a percentile. A score of 90 is most definitely NOT the 90th percentile.

You'd think they would have looked into that first.

 

[sohsie]

Everyone knows the two true predictors of board failure:

1) Wearing scrubs
2) Listening to music on headphones at work

[/end sarcasm]

Does anyone have a link or can post the text of the abstract?

 

[yaah]

Yeah, interesting abstract - thanks for mentioning it. I hadn't seen it yet.

I don't think you can link to the abstracts - they are online but it is password protected, I think anyone can register there, you don't have to be a USCAP member. http://www.abstracts2view.com/uscap10/index.php . It is an interesting abstract though - I know everybody generally only cares about the 3 digit score but the two digit score is probably more standardized year to year. And yes, it is not a percentile. They don't really say in the abstract whether they are using the real two digit score or they are figuring out the percentile. Might make a difference. And I agree 90 is not a terribly high score but it is high enough to easily pass the test.

Here is the conclusion:

Conclusions: At the UPMC pathology program, when residents scored &#8805;90 on the USMLE Step 1, there was a zero percent ABP failure rate, but there was a significant first-time taker ABP failure rate when the USMLE percentile score was &#8804;80. US medical graduates with USMLE percentile scores 81-89 had the highest ABP failure rate (35.7%), compared to other groups, including IMG graduates, when USMLE percentile score was <90 (Table 1). We consider this performance due to the fact that 24% of IMG graduates had at least two years of previous IMG pathology training. This data indicates that USMLE Step 1 percentile scores &#8805;90 and &#8804;80 are strong predictors of ABP first time failure rate in the UPMC program, while scores of 81-89 are less predictive.

Vaguely related--did you see the paper in Modern Pathology about the outcome of abstracts presented at USCAP? Only 30-40% are ever published.

Was that a paper? I thought that was another one of the abstracts. Would be very humorous if that never made it to publication as a paper. I am not surprised though - a lot of the abstracts are not going to pass muster as papers, or they are simply part of data that goes into a different paper.

 

[2121115]

I didn't have to log in so I don't think they are password protected. Here is the link...

http://www.abstracts2view.com/uscap10/view.php?nu=USCAP10L_551


Of note, this abstract is nominated for a Stowell-Orbison Award so I think that means it was actually written by a housestaff member at UPMC.

 

[spongioform]

Somewhat off-topic but I'm wondering how much time during residency is designated or available for study?

Also, about this statement above, "The correlation is not surprising - good test takers tend to remain good test takers even when the test gets more difficult."
I think it has less to do with taking tests than possessing the knowledge to answer the questions correctly. As you advance chronologically in testing, from SAT-->MCAT-->USMLE 1, the component of innate ability to take tests becomes less pronounced and there's a much larger dependence on actual knowledge possessed. At least this has been true in my experience.

 

[pathstudent]

it looks like practical information for program directors. If you know a candidate was likely to fail boards, you would rank them low on the rank list.

IN 2009 the first time pass rate was 88% for AP and 82% for CP so it seems like the test is getting more lenient, that or more intelligent people are going into path than those taking it in the early 00s

 

[lipomas]

I just looked at the abstract.

FAIL.

They are reporting the 2 digit score as percentile. The 2 digit score is NOT a percentile. A score of 90 is most definitely NOT the 90th percentile.

You'd think they would have looked into that first.

lol, I didn't notice that. I wonder if they actually mean two digit score then, or if they actually went through the trouble of figuring out the percentile from the three digit score and other data?

While this correlation is not surprising I think people tend to downplay such correlations or deny that they exist. I have heard people say, "doing well on the USMLEs or RISEs has no bearing on whether you will pass the boards." This is more evidence that that isn't true.

 

[green mantis]

This is rather interesting information. It raises many questions that would be nice to follow up on.

1) Is there a reason why IMGs have a higher passing rate?
2) Did they take 1st-time taker scores for the USMLE? I know of people taking them more than once.
3) How do scores for other parts of the USMLE equate to board passing (USMLE Step 3)?
4) They also mention 2 residents w/ USMLE equivalent scores. I'm guessing this is the COMLEX. I'm a DO, & I've never heard of programs considering the 2 exams as equivalent.

I doubt it's a problem to publish your program's pass rates, considering they're 10% higher than the national average. Shouldn't this kind of information be available for all programs?


----- Antony

 

[yaah]

IN 2009 the first time pass rate was 88% for AP and 82% for CP so it seems like the test is getting more lenient, that or more intelligent people are going into path than those taking it in the early 00s

I would say the latter, plus people are getting better at studying for it. I don't think the test is getting easier.

 

[pathstudent]

I would say the latter, plus people are getting better at studying for it. I don't think the test is getting easier.

How would you know unless you took it multiple times or had complete remberances from post tests

 

[BrainPathology]

1) Is there a reason why IMGs have a higher passing rate?

Not to be simplistic or to offend everyone but my first thought was that 4 years of residency would do wonders for your english if you didn't learn it way before starting medicine. This would definitely make passing easier. Also it's possible overseas schools teach to slightly different subject matter than the USMLE? (wasting money in new and exciting ways is the standard of care in nearly every institution besides Mayo and Cleveland Clinic and I doubt that's taught so much elsewhere) Maybe learning the subject matter in a residency which is tested on the boards is a more fair indicator of your skills.

 

[pathstudent]

This is rather interesting information. It raises many questions that would be nice to follow up on.

1) Is there a reason why IMGs have a higher passing rate?



----- Antony

Because the speak English gooder than us.

 

[BrainPathology]

Because the speak English gooder than us.

Outside of professionally degreed populations this is true nearly 100% of the time actually.

 

[yaah]

How would you know unless you took it multiple times or had complete remberances from post tests

Because it was a really hard test and I have talked to dozens of people who have taken it in the past 15 years, as well as people on the board and who write questions for it.

 

[exPCM]

In spite of the flaws in this study I can see this type of thing gaining traction among some PDs. I know of fellowship programs in path that use USMLE scores as a major factor in weeding out applicants, particularly in dermpath where the derm residents that apply usually have done well on the USMLE.

 

[2121115]

Residencies already use the USMLE to weed out residency applicants so I don't see why fellowship would be any different. A 2 digit score of 90 is about the national average (3 digit = 220ish). The average Step 1 score for pathology as a whole is roughly equal to the national average. Therefore, it isn't a big stretch to say that if your Step 1 is below the national mean then it is also below the mean for pathology applicants and therefore that applicant is more likely to not pass the boards.

The interesting part is that the pass rate for the boards is in the mid-high 80% (if I remember correctly?) for first time test takers. So theoretically being even in the 3rd quartile for Step 1 (which is below the mean) should still correlate well with passing the AP/CP boards. However, based on this abstract that doesn't appear to be the case. This may be accounted for by variation in training programs. Maybe a sign of a good program is whether or not they can take people with borderline Step scores and get them all to pass the boards. By contrast, it doesn't take much teaching for a program to take people who were always going to pass anyway (and may have passed even in 1st year based on test taking skills and prior knowledge) and get a 100% board pass rate.

 

[Adrian Cocot]

This just in: People who are good test-takers in medical school continue to be good test-takers in residency.

That's some fine research there, Lou.