Predicting USMLE score from COMLEX score

[group_theory]

I thought this would be an interesting article to discuss for journal club. In true journal club fashion, skim through the article at the last minute, and criticize it, tear it apart. What are the strengths? What are the weakness? Do you agree with the conclusion? Does the statistics support the conclusions? How would you have done the study? What followup study should be done?

Main question that every poster should answer: Would you provide this formula to ACGME Program Directors who are unfamilar with COMLEX scores?

A summary is provided below; please see the Sept 2006 edition of the JAOA for full article (http://www.jaoa.org/cgi/content/full/106/9/568)

Slocum PC, Louder, JS. How to Predict USMLE Scores From COMLEX-USA Scores: A Guide for Directors of ACGME-Accredited Residency Programs. J Am Osteopath Assoc. 2006;106:484-485

Objective: To evaluate the correlation between scores on COMLEX-USA and USMLE by COM students who took both examinations

Methods: Scores were analyzed from a cohort of COM students who took COMLEX-USA Level 1 and USMLE Step 1 and a cohort of students who, during the same time, took COMLEX-USA Level 2 and USMLE Step 2.

Results: At the Kirksville (Mo) College of Osteopathic Medicine of A.T. Still University of Health Sciences, 155 students took both COMLEX-USA Level 1 and USMLE Step 1 and another cohort of 56 students took COMLEX-USA Level 2 and USMLE Step 2. The Pearson product moment correlation of the scores from COMLEX-USA Level 1 and USMLE Step 1 was 0.83 (P<.001) and for COMLEX-USA Level 2 and USMLE Step 2 was 0.70 (P<.001). Scores on USMLE was predicted by corresponding COMLEX-USA scores using the equations USMLE Step 1 = 67.97 + 0.24 x COMLEX-USA Level 1 (R^2 = 0.68); and USMLE Step 2 = 102.2 + 0.18 x COMLEX-USA Level 2 (R^2 = 0.46).

Conclusions: A method of estimating USMLE scores from COMLEX-USA scores is provided for residency directors of institutions accredited b y the Accreditation Council of Graduate Medical Education for the evaluation of COM applicants.

---

Members don't see this ad.

 

[FutureNavyDOc]

With a n=56 DO students at 1 school, I think the validity of this study is virtually nil.

 

[surrender903]

good thing i wasnt in that study...

*rolls eyes*

n = 56, give me a break..

 

[FutureNavyDOc]

good thing i wasnt in that study...

*rolls eyes*

n = 56, give me a break..

I would also like to hope that it is not accurate since a score of 430 on COMLEX (roughly 1 SD below the mean) is equivalent to a 171 according to this study, with 182 being the current minimum passing score.

The mean COMLEX score of 500 is only a 187 on the USMLE while the mean on the USMLE is in the 224 neighborhood.

Also, the minimum passing score of 182 on USMLE would be equivalent to a 475 on COMLEX, so somewhere in the neighborhood of 30% of DO students would fail the USMLE.

Does anyone else find this bothersome?


Again, I doubt the accuracy of this study very much due to the small, similar, non-random sample.

 

[Krazykritter]

good thing i wasnt in that study...

*rolls eyes*

n = 56, give me a break..

NavyDoc,

What were you thinking?? Discussing the merits of a research study based upon its statistical evidence?

This is the problem w/ the JAOA & why it won't ever make it up to par w/ JAMA or other respected journals. A paper w/ this ridiculously small sample size should be laughed at & taken w/ a grain of salt instead of published. Oh wait though...we aren't supposed to analytically evaluate articles according to surrender903. Whoops.

 

[FutureNavyDOc]

NavyDoc,

What were you thinking?? Discussing the merits of a research study based upon its statistical evidence?

This is the problem w/ the JAOA & why it won't ever make it up to par w/ JAMA or other respected journals. A paper w/ this ridiculously small sample size should be laughed at & taken w/ a grain of salt instead of published. Oh wait though...we aren't supposed to analytically evaluate articles according to surrender903. Whoops.

Easy big fella, this study had 8 times the sample size of many of the Cranial research articles that get published!

 

[Taus]

I would also like to hope that it is not accurate since a score of 430 on COMLEX (roughly 1 SD below the mean) is equivalent to a 171 according to this study, with 182 being the current minimum passing score.

The mean COMLEX score of 500 is only a 187 on the USMLE while the mean on the USMLE is in the 224 neighborhood.

Also, the minimum passing score of 182 on USMLE would be equivalent to a 475 on COMLEX, so somewhere in the neighborhood of 30% of DO students would fail the USMLE.

Does anyone else find this bothersome?


Again, I doubt the accuracy of this study very much due to the small, similar, non-random sample.

well...the nbme data from the last few years does show that b/w 70-74% of DO's pass the USMLE....so oddly enough...this does work out...

 

[surrender903]

NavyDoc,
A paper w/ this ridiculously small sample size should be laughed at & taken w/ a grain of salt instead of published. Oh wait though...we aren't supposed to analytically evaluate articles according to surrender903. Whoops.

no more sarcasm for you i suppose...

 

[Brooklyndo2.1]

please also lets not forget, the usmle is a test, the comlex is a crapshoot

 

[group_theory]

Ahh, some discussion ... good

The article is still not available online.

Anyway, from the article

"the mean +/- SD scores were as follows:
COMLEX Level 1 - 549 +/- 73
USMLE Step 1 - 201 +/- 22
COMLEX Level 2 - 575 +/- 86
USMLE Step 2 - 200 +/- 22"


There are several things wrong with this study
The one mentioned already is the small sample size. But there was no discussion on statistics. Did they determine how many subjects must be enrolled into the study in order for the study to be predictive and statistically valid? Why just one year? Why not go back several years? Why not make it a multi-centered study and enroll the help of other schools?

What was the time frame that the students use to take comlex and usmle. Were they taken in a relatively same amount of time, or was there a lag? Was there a statistical difference between those who took it at the same time and those who did not? How did the students who only took COMLEX compare to students who took COMLEX and USMLE?

The formula for COMLEX 1 --> USMLE 1 has an R-square of 0.68 and the formula for COMLEX 2 --> USMLE 2 has an R-square of 0.46. Yet the comment section says "A COM student's COMLEX-USA Level 1 and Level 2 scores can reasonably and accurately predict USMLE Step 1 and Step 2 scores. Our formulas provide a method of comparison for residency directors in ACGME-accredited institutions as they evaluate applicants from COMs". With these R-square values, would you support that statement? Would you give your ACGME program directors these formulas with these R-square.

Or should we focus on the Pearson product moment correlation coefficient instead? For Step 1, it was 0.83 (P<0.001) and for Step 2, it was 0.70 (P<0.001)

 

[Krazykritter]

Good point about the R-square values...I didn't even look at that.

Superficially it looks really good when they state the p-values for the Pearson. It just seems that they are overlooking a whole lot experimental design flaws & poor R-square values to just throw some decent p-values in our faces.

 

[CatsandCradles]

The mean COMLEX score of 500 is only a 187 on the USMLE while the mean on the USMLE is in the 224 neighborhood.

Also, the minimum passing score of 182 on USMLE would be equivalent to a 475 on COMLEX, so somewhere in the neighborhood of 30% of DO students would fail the USMLE.

Does anyone else find this bothersome?

A 20-30% fail rate doesn't suprise me. In Norman Gevitz's book, he has the USMLE pass rates by DOs and its at about 77.0%

I know it sucks, but then again some schools aren't allowing people time to study for the USMLEs- or so I heard.

 

[FutureNavyDOc]

well...the nbme data from the last few years does show that b/w 70-74% of DO's pass the USMLE....so oddly enough...this does work out...

I would imagine that in general, the DO students who take the USMLE are on average smarter/ have better grades than those who take only the USMLE.

Even if we take those DO students who take the USMLE to be representative, doesn't it bother anyone else that 26-30% of DO students can't pass the USMLE Step 1?

Even if we assume 10% of DO's fail COMLEX Step 1, and that same 10% would also fail the USMLE, this means that at least 16-20% of DO's aren't smart enough/prepared enough to be physicians on level with MD's in terms of test scores. I can't imagine any non-off shore MD schools would be proud of a 16-20% failure rate.

Does this bother anyone else?

Is this indicative of a flaw in our education process? Is it indicative of a flaw in the DO admissions process? Or does it mean absolutely nothing and I'm just paranoid?

 

[CatsandCradles]

Here's the USMLE link that gives you the break down of allopathic vs osteopathic students taking the USMLE. http://www.usmle.org/scores/medlic.htm

I would imagine that in general, the DO students who take the USMLE are on average smarter/ have better grades than those who take only the USMLE.

Agreed.

Even if we take those DO students who take the USMLE to be representative, doesn't it bother anyone else that 26-30% of DO students can't pass the USMLE Step 1?

I'm sure there are idiots with 19 MCATs who are taking the USMLE and they're pulling down the scores. Further more, if you look on each year on the link above, there aren't too many DOs who actually take the USMLEs. Only 823 vs 17,000.

The only section where we might score higher than allopathic students is in Step 2 or 3 - but that's only because very few of us take Step 2 and 3.

Even if we assume 10% of DO's fail COMLEX Step 1, and that same 10% would also fail the USMLE, this means that at least 16-20% of DO's aren't smart enough/prepared enough to be physicians on level with MD's in terms of test scores. I can't imagine any non-off shore MD schools would be proud of a 16-20% failure rate.

I think off shore medical schools aren't doing too well. I will check around.

Does this bother anyone else?

Is this indicative of a flaw in our education process? Is it indicative of a flaw in the DO admissions process? Or does it mean absolutely nothing and I'm just paranoid?

It concerns me just as well.

 

[Taus]

I would imagine that in general, the DO students who take the USMLE are on average smarter/ have better grades than those who take only the USMLE.

Even if we take those DO students who take the USMLE to be representative, doesn't it bother anyone else that 26-30% of DO students can't pass the USMLE Step 1?

Even if we assume 10% of DO's fail COMLEX Step 1, and that same 10% would also fail the USMLE, this means that at least 16-20% of DO's aren't smart enough/prepared enough to be physicians on level with MD's in terms of test scores. I can't imagine any non-off shore MD schools would be proud of a 16-20% failure rate.

Does this bother anyone else?

Is this indicative of a flaw in our education process? Is it indicative of a flaw in the DO admissions process? Or does it mean absolutely nothing and I'm just paranoid?

Could it be possible that the different schools "teach to" the individual tests....ie Molecular/Biochem/Genetics are things that aren't really on the COMLEX and although we learn them, we don't harp on them more then we have to if they're not directly linked to a topic in pathophys and they won't be repeated in school-wide board review sessions? It also probably isn't common practice for a DO school to administer NBME (or similar) exams at the end of 2nd year and make those USMLE materials readily available.

I don't have an answer to any of this....just thinking...... I also consider myself "informed" about the test differences through SDN, whereas I know many classmates will go in and take the USMLE after the COMLEX without knowing about the differences and preparing for them specifically.

In the end...be informed of what you need to do and tear your boards a new a$$hole...

 

[allendo]

Could it be possible that the different schools "teach to" the individual tests....ie Molecular/Biochem/Genetics are things that aren't really on the COMLEX and although we learn them, we don't harp on them more then we have to if they're not directly linked to a topic in pathophys and they won't be repeated in school-wide board review sessions? It also probably isn't common practice for a DO school to administer NBME (or similar) exams at the end of 2nd year and make those USMLE materials readily available.

I don't have an answer to any of this....just thinking...... I also consider myself "informed" about the test differences through SDN, whereas I know many classmates will go in and take the USMLE after the COMLEX without knowing about the differences and preparing for them specifically.

In the end...be informed of what you need to do and tear your boards a new a$$hole...

comlex has more bugs n drugs from what I've heard! Usmle you better know your biochem and molec bio. I don't think alot of D.O. students practice for the usmle, by taking the nbme test and doing usmle q banks.

 

[Stimulate]

Does anyone know where I can get a copy of the full article. My residency program has a full-time statistician so I can have him review the article if I can get a copy.

I am not great at statistics but one thing that has always bothered me is why is the standard of deviation so enormous on the comlex exams? When you are looking at using a formula, every value you input into that formula must be accurate or there will be major defects in the calculation.

My gut feeling is that this topic has not been studied for a reason. I hope this is not true, but for example, the osteopathic medical schools have a ton of data on COMLEX scores and USMLE scores such that a large multi-center study could easily be undertaken.

However, what if we design a perfect study and come up with an accurate formula, and that accurate formula predicts the average COMLEX-taker would fail the USMLE (i.e. the 171 score mentioned above). When/if that study comes out, then there may be a big issue with proving the validity of COMLEX as an examination for licensure. I heard rumors in the past that the validity of COMLEX 3 was weak at best.

Anyway, imagine the public image nightmare that would occur if the average DO was being licensed when an MD with similar scores would not be allowed to practice. Yikes. I don't think the AOA answer of "but they are totally different tests" wouldn't hold up in public opinion.

My personal feeling about the COMLEX has always been that there should be 1 pathway to licensing in the United States and it should be the USMLE. I feel it should be USMLE because both MDs and DOs can qualify to take the USMLE, and USMLE is a much better written exam. The level of eloquence in the questions is amazing. When I took USMLE I always knew exactly what the author was testing (even though I didn't always know the answer...hehe) but with COMLEX on many questions I didn't even know what knowledge the author was testing or even what they were really asking.

If COMLEX were eliminated the AOA can always mandate that there be a separate test for OMM that is required to graduate from medical school (in fact they could use this stupid clinical skills exam that the younger generations are getting screwed with).

In the long-term, it would be far better for DOs to be able to say that we have the same education process and take the same licensing exams as MDs.

 

[group_theory]

Here is the link to the article
http://www.jaoa.org/cgi/content/full/106/9/568

(Edited my first post to include link now that it is available online)


While USMLE is the licensing exam for MDs and COMLEX is the licensing exam for DOs, the field to determine whether COMLEX is a good licensing exam does not belong to statistics but psychometric. It is psychometrics that determine that the USMLE is adequate in determining minimum competence for medical licensing, and it is psychometrics that determined that COMLEX is adequate in determining minimum competence for osteopathic medical licensing. To then compare COMLEX to USMLE using statistics, while on the surface seems like a good idea, runs into problems in the field of psychometrics.

A few years back, the Federation of State Medical Boards commissioned a study with the cooperation of the NBME and the NBOME. An independent person with a PhD in psychometrics (and pre-approved by the NBME and NBOME) was commissioned to study the validity of the COMLEX-USA series as a valid licensing examination for osteopathic medical physicians. He also looked at the USMLE to see if it was valid in determining licensure for allopathic physicians. The conclusions were that the COMLEX-USA and USMLE were valid examinations for osteopathic and allopathic physicians respectively. His conclusion, along with the FSMB committee recommendation to accept COMLEX as a valid examination for licensure, is one reason why Mississippi (the last state that did not accept COMLEX but required USMLE) to amend its laws to accept COMLEX for licensure. (note, there were other factors involved which lead to the change, including the daughter of one of the Mississippi Board of Medicine members being a DO). Now all 50 states accept COMLEX as a valid licensure examination.

Since I don't know much about psychometrics and standardize testing protocols (and I suspect many people on SDN have little knowledge of how standardize tests are actually scored), I will have to defer the discussion of whether comlex is a valid test when compare to the usmle (even if there is a statistically valid formula to determine comlex score to usmle score).

Anyway, do have your statistician look at this paper ... i suspect that it is a weak study statistically and no valid conclusions can be drawn from it due to many issues, the poor R^2 values, the low sample size, etc. My guess is that there probably isn't enough information provided in the paper to show that it is or isn't statistically significant.

 

[Arch Guillotti]

I really doubt there is a strong correlation between COMLEX and USMLE scores. What does it really matter anyway? I have taken I/II/III of the USMLE and COMLEX I/II (skipped part III). To me there really wasn't any comparison between the tests, USMLE Step I was very difficult but at least the questions were very straightforward whereas the COMLEX questions often were very poorly written. Step & part 2 weren't so bad and neither was Step III. Of the 5 tests for licensure I have taken, I scored below the mean on USMLE Step I and above the mean on all the others. I studied relatively equal amounts of time for each test. Anecdotally, I took Step I with 2 classmates, and they both failed. It does not surprise me that a decent percentage of DO students flunk Step I, whatever that is worth.

 

[sether52do]

I would also like to hope that it is not accurate since a score of 430 on COMLEX (roughly 1 SD below the mean) is equivalent to a 171 according to this study, with 182 being the current minimum passing score.

The mean COMLEX score of 500 is only a 187 on the USMLE while the mean on the USMLE is in the 224 neighborhood.

Also, the minimum passing score of 182 on USMLE would be equivalent to a 475 on COMLEX, so somewhere in the neighborhood of 30% of DO students would fail the USMLE.

Someone can correct me if I'm wrong, but I think this is the case. 70% of osteo USMLE takers pass. And I heartily agree that DO schools should gear their education towards the USMLE, we should drop the COMLEX and take a 50 ques OMM test the next day. Problem(s) solved.

 

[FutureNavyDOc]

Someone can correct me if I'm wrong, but I think this is the case. 70% of osteo USMLE takers pass. And I heartily agree that DO schools should gear their education towards the USMLE, we should drop the COMLEX and take a 50 ques OMM test the next day. Problem(s) solved.

That idea makes way too much sense for the AOA to use it.

Great effort though!

 

[homeboy]

Someone can correct me if I'm wrong, but I think this is the case. 70% of osteo USMLE takers pass. And I heartily agree that DO schools should gear their education towards the USMLE, we should drop the COMLEX and take a 50 ques OMM test the next day. Problem(s) solved.

Ditto.
...but then again, how is the AOA going to maintain its claims that osteopathic education is so vastly different and deserving of its own test...

 

[homeboy]

this is totally irrelevant to board scores, but anyone notice how much the AOA is trying to push the "OMS" versus the "MS" nowadays?