What is the view of SDNers?

[JelloBrain]

---

Members don't see this ad.

Couldn't think of another title. Was reading something on Pubmed earlier today when this paper caught my eye.

http://www.ncbi.nlm.nih.gov/pubmed/21099388

In case the link doesn't work for everyone, here's the title and PMID:

"Are United States Medical Licensing Exam Step 1 and 2 Scores Valid Measures for Postgraduate Medical Residency Selection Decisions?" McGaghie WC, Cohen ER, Wayne DB. PMID: 21099388

The validity, or otherwise, of any standardized scores is something I have always questioned (despite being someone who gets high scores). I was wondering if this article might (hopefully) start to bring about a change in mindsets so that all residencies have balanced and happy people who (get to) do what they love, rather than settle for anything based on a single 3-digit number.

Any specialty where the residents truly enjoy what they do and find it rewarding will not have attrition and other work-life issues-your job would become your way of life-why would you ever stop doing something that gives you so much pleasure (my view)?

I know a lot of prejudice exists on these forums, for various reasons, and I am not even saying that the scores be completely overlooked. All I am saying is that the scores should not completely close the door for people with genuine love for a particular specialty all because of one test. This is a choice of the rest of your professional life. Even MCATs can be taken more than once to improve your score, but you cannot improve your USMLE scores! That seems quite the wrong set-up.

Just my views, what are yours?

 

[atsai3]

The validity, or otherwise, of any standardized scores is something I have always questioned (despite being someone who gets high scores). I was wondering if this article might (hopefully) start to bring about a change in mindsets so that all residencies have balanced and happy people who (get to) do what they love, rather than settle for anything based on a single 3-digit number.

Any specialty where the residents truly enjoy what they do and find it rewarding will not have attrition and other work-life issues-your job would become your way of life-why would you ever stop doing something that gives you so much pleasure (my view)?

I know a lot of prejudice exists on these forums, for various reasons, and I am not even saying that the scores be completely overlooked. All I am saying is that the scores should not completely close the door for people with genuine love for a particular specialty all because of one test. This is a choice of the rest of your professional life. Even MCATs can be taken more than once to improve your score, but you cannot improve your USMLE scores! That seems quite the wrong set-up.

Just my views, what are yours?

1. You are essentially arguing for a lower threshold of sensitivity for detecting high quality applicants, and maintaining a high threshold for weeding out low quality applicants. Evaluation data will always be noisy. The balance between false positives (mistakenly labeling applicants as high quality when they are not) and false negatives (mistakenly labeling high quality applicants as not high quality when they actually are) will always be at issue.

2. Maybe there should be a check box on the application form: "Check here if you loooooooove dermatology and if you believe that taking care of dermatologic diseases is your passion". Everyone who checks that box should be given an interview and detailed consideration for dermatology residency.

No. That is not practical.

3. In favor of a high false negative rate: Many program directors need to sort through large pools of applicants. They cannot carefully and critically evaluate each applicant and offer each applicant an interview. Maybe you were battling diarrhea on the day you took the medicine shelf exam, maybe you had a car emergency the week of your surgery oral exam, etc., and if they could just understand all these extenuating circumstances then they would really give you a fair shake. No. They just don't have the time and resources for that. The most competitive programs get large numbers of applicants. They can afford to use arbitrary screening criteria ("we do not accept anyone with a USMLE Step 2 score below 257") to whittle their applicant pools down to a manageable size.

4. Even if residency programs adopted a low threshold for detecting high quality applicants like you suggest, would this really make anybody happy? Don't you see the numerous posts on SDN where people complain about how the interview process is rigged, "I just don't interview well", "the process is biased against non-native English speakers", etc etc etc.? So one might expect that these applicants would like to do an away rotation to prove that they are diamonds in the rough. But even if they did, don't you see the numerous posts on SDN where people complain about how their grade on an away rotation was 'not representative' of their true quality, "how can the attending adequately assess me after seeing me for 15 minutes a day for just one month", etc etc etc.?

My interpretation of all these complaints about the current quality metrics is that some people just don't like being evaluated. They will always feel like they have some hidden, underlying quality to them that no one else can see. It is hard to argue with someone who adopts a stance like that, because by definition you can't detect what a high quality applicant they really are.

5. Your suggestion about offering USMLE do-overs wouldn't really help. Would I look favorably on an applicant who got a 190 on the first try, 220 on the second try, and 260 on the third try? Not really.

-AT.

 

[Arcan57]

Identifying qualified applicants for specialized jobs is difficult in every field. Even in pro football where the amount of resources spent on determining draft order verges on infinite, selecting who will be successful relies heavily on chance. While medicine is not as competitive as the NFL, it still represents a field where there is significant differences between levels. The qualities that make for excellence as a medical student do not overlap perfectly with the qualities that make for excellence in residency.

Given the imprecision involved in picking residents, relying on stellar standardized measures (USMLE for medicine, the combine for the NFL) while avoiding obvious red flags (poor rotation grades/disciplinary actions in medicine, injury/legal problems for the NFL) remains a solid heuristic. Ignoring USMLE scores would be akin to relying on a recruit's home program's assessment of their ability, which is likely to be biased. An honors at one program is may be a high pass at another, and I wouldn't put it past med schools to inflate grades to improve their match percentage for the more competitive specialties.

 

[CardiacIntensivist]

Couldn't think of another title. Was reading something on Pubmed earlier today when this paper caught my eye.

http://www.ncbi.nlm.nih.gov/pubmed/21099388

In case the link doesn't work for everyone, here's the title and PMID:

"Are United States Medical Licensing Exam Step 1 and 2 Scores Valid Measures for Postgraduate Medical Residency Selection Decisions?" McGaghie WC, Cohen ER, Wayne DB. PMID: 21099388

The validity, or otherwise, of any standardized scores is something I have always questioned (despite being someone who gets high scores). I was wondering if this article might (hopefully) start to bring about a change in mindsets so that all residencies have balanced and happy people who (get to) do what they love, rather than settle for anything based on a single 3-digit number.

Any specialty where the residents truly enjoy what they do and find it rewarding will not have attrition and other work-life issues-your job would become your way of life-why would you ever stop doing something that gives you so much pleasure (my view)?

I know a lot of prejudice exists on these forums, for various reasons, and I am not even saying that the scores be completely overlooked. All I am saying is that the scores should not completely close the door for people with genuine love for a particular specialty all because of one test. This is a choice of the rest of your professional life. Even MCATs can be taken more than once to improve your score, but you cannot improve your USMLE scores! That seems quite the wrong set-up.

Just my views, what are yours?

The situation you present sounds great but this is the real world and people lie. How would you expect to truly know who loves what specialty? The idea of weeding people out sounds terrible but the fact is we need a way to tell who is smarter, more hardworking and willing to fight for the position.

 

[McGillGrad]

There has to be an universal measure that provides the initial cut-off (i.e. USMLE scores). That allows for the program to interview a reasonable number of applicants for their finite spots.

The interview is the game-changer that weeds out those who obviously would not make a good fit. As difficult as it may be to admit, we are generally replaceable cogs in a machine during residency. Your 'performance' during the interview days conveys your enthusiasm and your polished character. That combination usually says a great deal about the "real" you.

The sheer numbers need to be culled and an objective exam tends to be the lesser of evils.

 

[2012mdc]

There has to be an universal measure that provides the initial cut-off (i.e. USMLE scores). That allows for the program to interview a reasonable number of applicants for their finite spots.

The interview is the game-changer that weeds out those who obviously would not make a good fit. As difficult as it may be to admit, we are generally replaceable cogs in a machine during residency. Your 'performance' during the interview days conveys your enthusiasm and your polished character. That combination usually says a great deal about the "real" you.

The sheer numbers need to be culled and a objective exam tends to be the lesser of evils.

It would get really messy without a universal measure. Grade inflation would run rampant and it would be all about connections. And step scores do have some correlation to residency board performance

 

[Law2Doc]

...And step scores do have some correlation to residency board performance

Well "some correlation" doesn't mean impressive correlation, but more importantly, passing boards is only one component program directors look for in selecting residents, and it's not the component that gives program directors the most headaches if they get it wrong. Lots of IMGs, for example, each year have unbelievably high USMLE numbers but the reason IMGs don't do so well in the match is because many don't fair so well in making the transition to the US clinical setting. First and foremost, the program director is choosing someone who can do the job of resident well enough that the attendings are happy, the patients are happy, and he doesn't have to hear about X resident screwing up this or not doing that. He wants the resident he chooses to not cause him headaches and not generate any complaints/issues over the 3-5 years he is working under the program. Doing well on in-service exams and passing the boards is important, but for the most part secondary. So things like good references, a strong interview, and away/audition rotation performance looms large in the process.

We on this board all know bad residents. They are not bad because they don't test well. In fact some of the worst residents are good standardized test takers.

But you have to realize that when many hundreds of people are applying for a small number of interview slots for an even smaller number of residency slots, something has to be used to make cuts. You cannot rely on med school grades because it's not a universal standard -- how do you compare a place where folks really work hard to get the same grades as another place that hands out high passes like candy? How do you compare a place that is true pass/fail in the first year to another place that uses a letter grade system? How do you trust a med school's grades when there is an incentive to be able to tell premeds that their graduates match into the top programs? You can't. So you have the USMLE scores, which are uniform. You perhaps have the shelf exams which are uniform. And you have the interview/away rotations where you can see applicants for yourself.

Choosing a resident who can do the job of resident is tough because the info you have to look at isn't exactly the info you need. But you definitely have to make do because there isn't currently anything better out there. So what you do is use the standardized tests to cull the herd into numbers you can handle, and then from that group, you interview and look at evals and LORs from faculty you trust, and try to discern which of the folks who scored well will work. Will someone who is a crummy test taker not get a shot because of the screening? Absolutely. But do residencies find enough people who both can score competitively and can wow them in the interview? Yes.

So there is no need, from the PD's perspective to change the system. They acknowledge that some good residents fall through the cracks, but you make some sacrifices to get a system that otherwise works. Until they change the USMLE so that you cannot use it in this way, in which case some other system will arise which will have its own flaws.

 

[DoctwoB]

Is the USMLE perfect? No.

Is it necessary? Yes.

 

[Law2Doc]

Is the USMLE perfect? No.

Is it necessary? Yes.

Well, the "USMLE" itself isn't necessary (DO's get by with their own COMLEX for example). And a strong argument could be made that a shelf exam in the field most closely related to the student's intended specialty would actually be a lot more useful to PDs (ie for folks going into OB, look at the OB shelf, for folks going into surgery, look at the surgery shelf, etc). But some universal, med school independent, yardstick is necessary. I suspect once the USMLE is revamped such that its timing won't work well with residency decisions, some other standardized test(s) will fill the void. The system could be made a lot more useful in terms of program director's needs by making it specialty specific. But there will always need to be a universal, standardized component, because the incentives for grade/eval inflation by med schools is too great.

 

[dragonfly99]

I agree w/above.
I also think that if USMLE's are not used somehow to evaluate applicants, the process of matching would be even more of a mess. There's too much sucking up alreadly in medical school, and without ANY objective measure to compare students I am sure it would be worse.

 

[turquoiseblue]

Couldn't think of another title. Was reading something on Pubmed earlier today when this paper caught my eye.

http://www.ncbi.nlm.nih.gov/pubmed/21099388

In case the link doesn't work for everyone, here's the title and PMID:

"Are United States Medical Licensing Exam Step 1 and 2 Scores Valid Measures for Postgraduate Medical Residency Selection Decisions?" McGaghie WC, Cohen ER, Wayne DB. PMID: 21099388

The validity, or otherwise, of any standardized scores is something I have always questioned (despite being someone who gets high scores). I was wondering if this article might (hopefully) start to bring about a change in mindsets so that all residencies have balanced and happy people who (get to) do what they love, rather than settle for anything based on a single 3-digit number.

Any specialty where the residents truly enjoy what they do and find it rewarding will not have attrition and other work-life issues-your job would become your way of life-why would you ever stop doing something that gives you so much pleasure (my view)?

I know a lot of prejudice exists on these forums, for various reasons, and I am not even saying that the scores be completely overlooked. All I am saying is that the scores should not completely close the door for people with genuine love for a particular specialty all because of one test. This is a choice of the rest of your professional life. Even MCATs can be taken more than once to improve your score, but you cannot improve your USMLE scores! That seems quite the wrong set-up.

Just my views, what are yours?

I agree that usmle scores shouldn't determine your fate for the rest of your life or prevent one from getting into residency. I truly think it is unfair, especially because you are only allowed one chance and it preserves your fate in stone. if we could retake tests, I probably wouldn't have that big of a problem with it. And for someone that passed with 76's, or lower than 80, that nearly throws them out of a career forever, especially if they are IMG's.
They should raise the passing score to 80 if they want to make 80+ as a sole criteria. that way we could retake it until we are good enough.

I know that a lot of SNDer's disagree and want to keep the system as is as if it is so perfect, but i wholeheartedly disagree with that view.

 

[Arcan57]

I agree that usmle scores shouldn't determine your fate for the rest of your life or prevent one from getting into residency. I truly think it is unfair, especially because you are only allowed one chance and it preserves your fate in stone. if we could retake tests, I probably wouldn't have that big of a problem with it. And for someone that passed with 76's, or lower than 80, that nearly throws them out of a career forever, especially if they are IMG's.
They should raise the passing score to 80 if they want to make 80+ as a sole criteria. that way we could retake it until we are good enough.

I know that a lot of SNDer's disagree and want to keep the system as is as if it is so perfect, but i wholeheartedly disagree with that view.

There needs to be a standardized system, and that standardized system will always have cut-offs. Because of that, there will always be people that fall below that cut-off.

I don't know of any medical that de-emphasizes the importance of performing well on the USMLE Step 1. I don't know of any US medical school that does not allow time for preparation for the USMLE exam. The USMLE content is a relatively known quantity and there are a large number of high quality preparation material designed to fit almost every learning style. If, with all of these advantages, you are not able to obtain a score that will allow you to match into even one field of medicine then perhaps the problem is not with the test.

If there are extraordinary circumstances that would have caused your results to be not representative of your intellectual capacity, you are allowed to reschedule the USMLE exam. If you are arguing that you may need to take the test 2-3 times to achieve an adequate score, then I wonder why you did not adequately prepare for the first time. That would speak to a lack of judgement and maturity that are perhaps more troubling characteristics in a resident than simply doing poorly on the exam.

 

[JelloBrain]

Sorry for starting something and not having had a chance to respond so far. But views of people are as I had expected (down to individual posters).

I completely agree that there should be something that is standardized for residency selection. I am not against the USMLE per se. However I am not in favor of one exam being "set in stone" where a lot of applications are not even read as they are filtered based on just the Step 1 scores. Especially when the majority of what is tested on Step 1 has limited clinical application IMO.

I am not arguing in favor of people who cry foul at every thing, and seem to have a very hidden quality that, well, remains hidden. I was mainly advocating for those who really have one "less than ideal" day. Reviewing people's track records of school, undergrad, medical school performance can differentiate between the ones who are consistently poor performers and the ones who didn't perform as well on the one test that could make or break their career.

Atsai3-If you really think that people can improve their scores so much by retaking their test-190, 220, 260, then it is not a very reliable test, is it?

Also, do you all really believe that saying "I love some specialty (or someone)" is to be taken at face value? Or would you rather see how actions back up the said love-attending Sub-Is and doing well, doing research in the field, already trained in the field (for IMGs).

I wouldn't say "I looooove Dermatology", or any ROAD specialty for that matter (nothing against them, just not my thing), even if I was being offered interviews in them without asking. You can only have passion for one specialty, so surely you cannot lie-at the end of the day, you are lying to yourself, and no one else in the process.

So all those who are saying that people will lie to get into a particular fancy specialty, think about what is the use of lying-you'll have to live a lie for the rest of your life, or quit and move to a different specialty.

Law2Doc-I completely agree with your idea of a shelf exam in a person's chosen specialty. This would be beneficial to IMGs as well as majority are pre-trained in their chosen specialty, so taking these exams would be a lark for them. This can be used in addition to USMLE scores and other evaluations in determining one's interest and "fit" in a specialty.

And do medical schools really inflate grades? How does that benefit anyone?

Standardized tests and cut-offs are a necessity. But each program in different specialties using their own "cut-offs" and using it as an absolute measure of who gets face-time seems an inappropriate use of the reason these tests were designed in the first place (for state medical board licensure requirements).

What will happen when the USMLEs become pass/fail with no scores as I had heard might happen sooner rather than later? Then surely, track record will matter, as what might be a "weeding out" method to be used then?

 

[gutonc]

What will happen when the USMLEs become pass/fail with no scores as I had heard might happen sooner rather than later? Then surely, track record will matter, as what might be a "weeding out" method to be used then?

If/when this happens, the screen will then be more than likely be "AMG vs. not." So your 250/250/250 and 4 years as an attending orthopedic surgeon in Wherever-stan will be meaningless. And the Carib kids will be SOL.

 

[Arcan57]

And do medical schools really inflate grades? How does that benefit anyone?

It allows the school to state they have a track record of placing students at prestigious residencies and in difficulty to obtain specialties. It benefits the students who get a higher grade and thus make it easier to obtain their desired residency spot. The only really negative impact is on good students at other schools that earned a "high pass" that would have been an "honors" at the first school.

It's interesting that pass/fail evolved to replace the letter system of grading, yet there are now analogues to every grade. A=honors, B=high pass, C=pass, F=fail. They exist because while still subjective, residencies have to have some measure other than that person managed to find two faculty members that thought they would be a good resident in specialty X.

In terms of making students take a shelf exam based on their chosen specialty to replace the USMLE, that sounds like an absolutely terrible idea.

1) They already do take a shelf exam and it's usually a significant portion of their rotation grade.
2) It has all the problems of "one day, one test" that the USMLE does.
3) Residency is for training medical students in a specialty, not to take people that already know the specialty and put them to work for x years. And I don't think any system that inherently favors previously trained IMGs is good for the US or the world.
4) It would make selecting a specialty an even earlier decision, especially for the subspecialties that usually are not provided until late in 3rd year or as a 4th year. The person that discovers their love of a specialty late will be at a even bigger disadvantage than currently against the student that knew they were going into specialty X from the first day even if they realized their is something else they like better.

 

[JackADeli]

...What will happen when the USMLEs become pass/fail with no scores as I had heard might happen sooner rather than later?...

Interesting thought. Maybe it is that things move really, really slow in medicine or maybe it is that this is somepropogated pipe dream held by every new medical school class. However, I took USMLE step 1 over 13 years ago and before I took it, everyone was talking about how it would "be pass/fail soon". I remember people in my class having all the same discussions and how "unfair it is" that "we will get a score but future classes will not be disadvantaged by a non-representative score because they will be pass/fail..."

If/when this happens, the screen will then be more than likely be "AMG vs. not." So your 250/250/250 and 4 years as an attending orthopedic surgeon in Wherever-stan will be meaningless. And the Carib kids will be SOL.

Yep. The stellar USMLE score is sometimes the last chance for a FMG or Carib grad. I think a pass/fail system will further disadvantage them. If that happens, we can expect to hear from all of them about how unfair things are and that they don't have the objective score that surely would show how excellent they are....😴

You can't have it both ways. There can not be two seperate systems for people depending on what you think works best for you.... "I don't test well" crowd gets what, Pass/Fail? then the "I am excellent and in carib school because life was unfortunate or something" crowd gets a score? I remember classmates from undergrad that didn't think they would score well in class. They chose the Pass/Fail option. Then, when they scored well, they went crying to the admin and professors to get a formal grade in place of their original choice of Pass/Fail.

 

[Substance]

In Canada there's no standardized test like the USMLE, and all schools are pass/fail. So, your residency match boils down to who you know, with research playing a semi-important role.

One day, one test isn't a very good measure of one's knowledge either. People do have bad days. I remember a thread about a guy who failed step 1 and then got a 260 on the rewrite. It can happen. We should be mitigating that.

Why not just have a bunch of smaller tests instead of just one big one, spread out over time? Aside from cost, I cannot see any downside.

And if you don't test well..then you went into the wrong field.

 

[DartosDO]

Why not just have a bunch of smaller tests instead of just one big one, spread out over time? Aside from cost, I cannot see any downside.

And if you don't test well..then you went into the wrong field.

If only there was a way to test med students with small tests over a certain amount of time in various subjects... oh yeah - it's called MED SCHOOL

 

[2012mdc]

In Canada there's no standardized test like the USMLE, and all schools are pass/fail. So, your residency match boils down to who you know, with research playing a semi-important role.

One day, one test isn't a very good measure of one's knowledge either. People do have bad days. I remember a thread about a guy who failed step 1 and then got a 260 on the rewrite. It can happen. We should be mitigating that.

Why not just have a bunch of smaller tests instead of just one big one, spread out over time? Aside from cost, I cannot see any downside.

And if you don't test well..then you went into the wrong field.

That is a very rare scenario.

And the shelf exams 3rd year as well as Step 2. give you the opportunity to make up for a lower Step 1.

A good compromise is to require shelf scores to be reported along with Step 1. I wouldn't necessarily be in favor of that but it would help the crowd that doesn't want so much placed on Step 1

 

[Arcan57]

That is a very rare scenario.

And the shelf exams 3rd year as well as Step 2. give you the opportunity to make up for a lower Step 1.

A good compromise is to require shelf scores to be reported along with Step 1. I wouldn't necessarily be in favor of that but it would help the crowd that doesn't want so much placed on Step 1

I'm not a huge fan of the shelf exams because the students that got the highest scores usually did so at the expense of actually learning medicine. Clearly there are exceptions, but time spent caring for patients is time not spent studying to get that extra 5% points.

 

[turquoiseblue]

pass/fail seems like a better alternative, much like the clinical skills assessments were. that is how it was originally before they suddenly started doing the criteria. the way it is i mess up and got a 76 and can't change it, and i can't get into any specialty, let alone a desired one, even though i passed. it doens't mean i know less, it just means im not a good test taker. i study hard and long and know my stuff in reality but on the test it is a different world. scores don't reflect a person's knowlege necessarily. not to mention, there are some real rude people that can become doctors. integrity and compassion mean nothing in this day and age, just scores. i think something is wrong with that.

where did you see that it will soon become pass/fail? are there going to be a multitude of residency spots? that seems how that can become possible.

 

[JackADeli]

If only there was a way to test med students with small tests over a certain amount of time in various subjects... oh yeah - it's called MED SCHOOL

Yep. I think the theme is, "it would be really, really nice to have an alternative that specifically works for me. Anything else is just unfair and not representative". I think there is a whole other thread somewhere similar to this train but referencing the injustice of the MCAT. It goes something like, "the MCAT forced me to the Caribean....".

...where did you see that it will soon become pass/fail?...

It is at least 13 year old gossip.

 

[2012mdc]

I'm not a huge fan of the shelf exams because the students that got the highest scores usually did so at the expense of actually learning medicine. Clearly there are exceptions, but time spent caring for patients is time not spent studying to get that extra 5% points.

I don't think it's necessarily at the expense of learning medicine, it's probably at the expense of getting more specialty driven experience. The student that studies instead of scrubbing into another case or spending extra time with a patient just because they can isn't necessarily learning less.

I'm just in the middle of my 3rd year but in terms of clinical experience, working on my H&P skills (the doing and the writing) and my assessment/plan is where I have learned the most.

 

[NotAProgDirector]

where did you see that it will soon become pass/fail?

The USMLE is in the process of reassessing itself, something that it does every 10 years or so. That process is currently ongoing, and can be reviewed on the USMLE website. In the very first draft, the USMLE comprehensive review committee recommended that the exam be changed to P/F. The outcry from PD's and from some students was quite loud. The current working document notes that the primary purpose of the USMLE is for assessing the ability for licensure, but that secondary uses (such as selecting students for residencies) is OK as long as it doesn't get in the way of the primary purpose. Hence, any chance that the USMLE is switching to P/F is dead.

The original plan included changing the USMLE to two steps instead of three. "Gateway A" was supposed to evaluate a student's ability to practice under supervsision, and "Gateway B" assess for independent practice. The current document still talks about these two assessment points, but seems to suggest that the current 3 step process will remain. There also was some discussion of dropping CS, but that also seems dead.