[Teaching and Learning in Medicine] SJT's and USMLE

[Mr.Smile12]

Advertisement - Members don't see this ad

Fairness of a Professionalism-Oriented Situational Judgment Test for Residency Selection in the United States: A Predictive Bias Analysis by Gender, URiM Status, and International Medical Graduate Status

Residency selection has traditionally emphasized cognitive measures such as United States Medical Licensing Examination (USMLE) Step 1, Step 2 Clinical Knowledge (CK), and clerkship grades, even though physician performance encompasses cognitive and noncognitive competencies. Situational judgment tests (SJTs) have emerged as a promising complement, demonstrating criterion-related validity and producing smaller subgroup score differences than cognitive measures. However, smaller mean differences do not establish fairness. The Standards for Educational and Psychological Testing distinguish between mean group differences and predictive bias. The latter occurs when an assessment systematically over- or underpredicts outcomes for certain groups regardless of true construct standing. Whether residency-focused SJTs demonstrate predictive bias has not been examined—an important gap, as unknown bias risks unfairness to applicants and could undermine efforts to build a more representative physician workforce.

Link to article: https://www.tandfonline.com/doi/abs/10.1080/10401334.2026.2715477

 

[Wistfulness]

SJTs should not be used in any selection process, and especially the residency selection process. Adding more random BS "selection factors" for applicants to provide isn't the solution to the issues program directors or applicants are facing. You need application caps, dummies.

SJTs like Casper/Preview are mickey mouse science and simply cannot function as advocates claim. Without litigating the validity of the constructs these exams claim to measure, there's an unresolvable paradox. These exams claim to measure non-cognitive attributes and are not knowledge-based exams. Any exam in which the examinee has agency to manipulate their response can be made into a knowledge-based exam. All you need is knowledge of the exam, the qualities the exam looks for in the response, and to shape your responses to contain those qualities. Given that these processes are very competitive, people are going to talk about what worked for them and what didn't. Eventually they will be gamed out once they're deemed to be of importance and lose all meaning... and that's assuming they had any meaning to begin with.

Simply look at these forums. We're not having deep conversations about what is the right or wrong thing to do in these test examples. The ethical dilemmas in these tests are a joke and of no serious importance. How have things gotten this far with zero skepticism?

 

[Mr.Smile12]

SJTs should not be used in any selection process, and especially the residency selection process. Adding more random BS "selection factors" for applicants to provide isn't the solution to the issues program directors or applicants are facing. You need application caps, dummies.

SJTs like Casper/Preview are mickey mouse science and simply cannot function as advocates claim. Without litigating the validity of the constructs these exams claim to measure, there's an unresolvable paradox. These exams claim to measure non-cognitive attributes and are not knowledge-based exams. Any exam in which the examinee has agency to manipulate their response can be made into a knowledge-based exam. All you need is knowledge of the exam, the qualities the exam looks for in the response, and to shape your responses to contain those qualities. Given that these processes are very competitive, people are going to talk about what worked for them and what didn't. Eventually they will be gamed out once they're deemed to be of importance and lose all meaning... and that's assuming they had any meaning to begin with.

Simply look at these forums. We're not having deep conversations about what is the right or wrong thing to do in these test examples. The ethical dilemmas in these tests are a joke and of no serious importance. How have things gotten this far with zero skepticism?

I am wondering longer-term. Casper has been used in Canada for over a decade. What do we know about the competencies of Canadian physicians and health professionals where Casper has been incorporated into their admissions processes? How can we measure whether the test really correlates to "better" professional practice, or is this effect neutralized by the realities of medical training when additional context drives decision-making? This is all psychometric stuff to me, but a longer-term effect would be good to measure. Is Canadian health care vastly improved since the introduction of Casper or MMIs?

 

[Wistfulness]

I am wondering longer-term. Casper has been used in Canada for over a decade. What do we know about the competencies of Canadian physicians and health professionals where Casper has been incorporated into their admissions processes? How can we measure whether the test really correlates to "better" professional practice, or is this effect neutralized by the realities of medical training when additional context drives decision-making? This is all psychometric stuff to me, but a longer-term effect would be good to measure. Is Canadian health care vastly improved since the introduction of Casper or MMIs?

They haven't improved meaningfully as far as I'm aware. At best Casper scores are correlated quite weakly with Honors grades and OSCE scores. Which are, to my understanding, subjective and can be based on how much people like you. My favorite part about OSCE scores is that their goal is to increase the objectivity of grading, while the biggest complaint about them is the subjectivity of grading. The jokes write themselves.

Why would we think that incorporating Casper/MMI into the admissions process would improve anything over the long term? What outcomes is it supposed to be correlated with? That would assert that things like empathy, self-awareness, fairness, motivation, and problem solving are well-defined, and generalizable constructs that are consistent over a long period of time. They just aren't.

My big issue with Casper, reading between the lines of their marketing, is they argue there's this widespread epidemic of unempathetic and unethical doctors. These bad doctors are so because of their bad nature and it was the responsibility of medical admissions to catch them because they just aren't suitable to be a doctor. It's not the abusive training model? It's not the work hours? It's not the EPIC popups? Maybe the people aren't the problem and it's maybe the system the people are working under?

Undergraduate institutions are sending you their best and you're turning them into obsessive compulsive weirdos with anxiety disorders. How did medical schools mess this up so bad?