ChatGPT can pass the USMLE

[the5thelement]

Is this the end? Or the beginning of the end? Or the end of the beginning?

---

Members don't see this ad.

 

[Merovinge]

ChatGPT got a 60% on Step 1 which is very borderline passing and I think often not passing but depends on the year.

A seperate program that was specially fed a lot of medical knowledge/information got around a 67% (I think?) and pretty easily passed the exam.

 

[comp1]

This seems to say a lot more about the USMLE than ChatGPT.

 

[littlefred]

AI driven medical care may lead to better outcomes, would be interesting to see how ChatGPT fares against community mental health treatment with regards using Hamilton depression scales etc.

The issue might be the quality of evidence we have for our treatments and how the AI will recommend treatments fed from those studies.

 

[calvnandhobbs68]

I mean the Steps are literally memorizing a bunch of facts and restating them in various ways in a multiple choice format...something you'd expect a narrow AI to be pretty good at as it was pre-trained on huge datasets some of them certainly relevant to the tests.

I think the freeform responses and ability to generate completely new material in response to open ended questions are actually much more fascinating than the ability of an AI to answer a multiple choice test, something you'd expect it to be pretty good at doing (probably actually better than it's doing right now).

 

[Candidate2017]

AI driven medical care may lead to better outcomes, would be interesting to see how ChatGPT fares against community mental health treatment with regards using Hamilton depression scales etc.

The issue might be the quality of evidence we have for our treatments and how the AI will recommend treatments fed from those studies.

ChatGPT will agree with 90-95% of people coming in wanting an autism or ADHD diagnosis, and have higher patient satisfaction scores.

 

[littlefred]

ChatGPT will agree with 90-95% of people coming in wanting an autism or ADHD diagnosis, and have higher patient satisfaction scores.

Is that any different than what some clinicians are doing?

All joking aside, the stimulant problem needs to be addressed from the top as it seems to be more of a systemic issue rather than a diagnostic issue.

 

[clausewitz2]

I mean the Steps are literally memorizing a bunch of facts and restating them in various ways in a multiple choice format...something you'd expect a narrow AI to be pretty good at as it was pre-trained on huge datasets some of them certainly relevant to the tests.

I think the freeform responses and ability to generate completely new material in response to open ended questions are actually much more fascinating than the ability of an AI to answer a multiple choice test, something you'd expect it to be pretty good at doing (probably actually better than it's doing right now).

I got it to generate a version of the fight between Hector and Achilles in the Iliad in hexameter, only Achilles is a house cat. It also suggested a new cat pun-related title for the work. I for one welcome our AI overlords.

 

[Merovinge]

This seems to say a lot more about the USMLE than ChatGPT.

Did the same for the MBA exam as well. I think it just says more about answering multiple choice test questions.

 

[Trismegistus4]

Not that they will do this anytime soon, but the solution is to make the USMLE more g-loaded. ChatGPT can't even get these basic math/logic questions right:

 

[FlowRate]

Is that any different than what some clinicians are doing?

All joking aside, the stimulant problem needs to be addressed from the top as it seems to be more of a systemic issue rather than a diagnostic issue.

I know this is tangential to the tread, but I keep thinking that our "leadership" (major academic centers/academicians, professional societies) seem to have been caught on the back foot, just like the rest of us, on this issue.

And the issue started pre-pandemic--we were noticing escalating ADHD eval requests for adults when I was in my 3rd year of residency (2 years pre covid.)

 

[mistafab]

Bro, if a computer with perfect memory and limitless access to medical textbooks couldnt pass that exam, I would say the programmers are worthless.