Worlds Highest USMLE Score by an A.I

[SSerenity]

Raw data is available on their website. Really incredible progress we are seeing.
Also linking it here:

interestingly, Mayo Clinic previously published a report that included ChatGPT 4 that scored ~ 80% on the exact same practice test. Since then, the performance has gone down significantly. This tracks with a lot of user's reporting that ChatGPT has been getting dumber. And some people even claim that GPT4 and 4o users are actually using 3.5 under the hood.

---

Members don't see this ad.

 

[LucidSplash]

If the AI looks at the total of what’s on the Internet, and there’s a lot more misinformation than accurate expert information, makes sense it would progressively come up with worse responses.

 

[Nontrad_FL_LGBT]

If the AI looks at the total of what’s on the Internet, and there’s a lot more misinformation than accurate expert information, makes sense it would progressively come up with worse responses.

Joint pain and a rash? Are you sure it isn't MCAS and EDS and POTS and DID? 😂

 

[LucidSplash]

Joint pain and a rash? Are you sure it isn't MCAS and EDS and POTS and DID? 😂

Chronic Lyme.

 

[voxveritatisetlucis]

All non surgical specialties will likely be replaced within 10 years. Why pay a physician 400k per year if an AI can do most of it for 30 per month. If you’re in these fields, taking out margin loans or mortgages to buy NVDIA, Microsoft, Google, Meta is prudent. If you’re in surgery, Intuitive Surgical and Stryker may be good hedges in case AI assisted robotics progresses faster than anticipated.

If you think that’s too short of a timeline, the iPhone only came out like 17 years ago. In 20 years, AI will surpass humans in everything

 

[RangerBob]

All non surgical specialties will likely be replaced within 10 years. Why pay a physician 400k per year if an AI can do most of it for 30 per month. If you’re in these fields, taking out margin loans or mortgages to buy NVDIA, Microsoft, Google, Meta is prudent. If you’re in surgery, Intuitive Surgical and Stryker may be good hedges in case AI assisted robotics progresses faster than anticipated.

If you think that’s too short of a timeline, the iPhone only came out like 17 years ago. In 20 years, AI will surpass humans in everything

Because computers don't have licenses to practice medicine.

Realistically, if the corporate powers that be haven't been able to get foreign docs licensed to practice diagnostic radiology (arguably the easiest specialty to outsource), they're not going to get AI licensed to practice medicine legally in any state anytime soon.

We see a few programs that now allowed foreign-trained MDs a pathway to practice here in the US, but most have such arduous requirements that realistically I don't think it'll make a huge dent. And that's still vastly different than the large scale outsourcing of diagnostic radiology that would occur if a US state license/board certification/etc weren't required and healthcare corporations/VC could do whatever they wanted.

Similarly, I don't think AI will take over any doc's jobs. Not anytime soon.

But AI will make our lives "easier" the same way EMRs have!

You also forget how incredibly slow technology is to enter modern medical practice. Everything has to go through the FDA. Less than 10 years ago intrathecal pumps had the ugliest programming devices (all B&W--it looked like the TI-83 calculator I had in high school) because they need to be FDA approved and take a decade or two to design/obtain approval.

I've said it before and I'll say it again--when AI can replace physicians, the world has vastly more problems to worry about. By the time we're replaced, accountants, bankers, programmers, lawyers, likely most engineers, etc., are all out of jobs.

 

[mrbreakfast]

All non surgical specialties will likely be replaced within 10 years. Why pay a physician 400k per year if an AI can do most of it for 30 per month. If you’re in these fields, taking out margin loans or mortgages to buy NVDIA, Microsoft, Google, Meta is prudent. If you’re in surgery, Intuitive Surgical and Stryker may be good hedges in case AI assisted robotics progresses faster than anticipated.

If you think that’s too short of a timeline, the iPhone only came out like 17 years ago. In 20 years, AI will surpass humans in everything

Bro all you post is stuff like this. I assume you're dropping out of medical school and putting your remaining tuition money into NVDA?


As for the OP, I have many thoughts, but will keep it to a few:
1) Med students and laypeople put massively more weight into "AI is killing it on Step 1" than they should; it's a test full of association and buzzwords, of course a program crushes it. That's literally nothing like real medicine.

2) AI in medicine will probably be more like chess engines in chess. For a few bucks, you can get a program for your laptop that can easily beat any human in chess, and that has been the case for probably 20+ years now. How has that translated into the game? Mostly for niche positional analysis. It's still human vs human.

3) As far as AI taking human jobs, this reminds me of a good med student quiz: "does the patient in front of you have Parkinson's disease, or do they not? You can run any tests or imaging studies you want." A trained human can answer this in a few minutes; even in 20 years I doubt AI will be able to.

Medicine will be affected by AI for sure. But for many reasons, it'll be one of the last fields to be affected.

 

[Sidus1011]

All non surgical specialties will likely be replaced within 10 years. Why pay a physician 400k per year if an AI can do most of it for 30 per month. If you’re in these fields, taking out margin loans or mortgages to buy NVDIA, Microsoft, Google, Meta is prudent. If you’re in surgery, Intuitive Surgical and Stryker may be good hedges in case AI assisted robotics progresses faster than anticipated.

If you think that’s too short of a timeline, the iPhone only came out like 17 years ago. In 20 years, AI will surpass humans in everything

Your feed is a literal continuous eyeroll. If we are out of a job, 95% of America is out of a job - at that point society will have to undergo massive restructuring beyond what is currently comprehensible. We will have bigger problems than some tech nerds trying to usurp a physician's role in patient care.

 

[SSerenity]

I think almost everyone is fixated on the wrong question.
It seems like most people in medicine react to A.I. by asking "Is this good enough to treat patients", "can it replace doctors" or some variation of that.

The real question you should be asking is, "how can I use A.I to be a superhuman doctor". Do you realize how fast A.I can parse through literature and guidelines to help answer questions? And I don't mean "diagnose this" questions, or USMLE style questions. I mean real nuanced questions that will help you make a decision, not make the decision for you.

I'm sure there is a small handful of super-human gunner students who have figured out an efficient way to use A.I in their study flow. I know when I work through practice questions, I have a ton of other questions that I need answered and just don't have time to go read up on each one, so I take what I can get in my allocated block.

A.I has the potential to create some insanely competent med students, thats the way I see it. Its the difference between "Figure this out for me" vs "help me learn this faster".

 

[SSerenity]

Bro all you post is stuff like this. I assume you're dropping out of medical school and putting your remaining tuition money into NVDA?


As for the OP, I have many thoughts, but will keep it to a few:
1) Med students and laypeople put massively more weight into "AI is killing it on Step 1" than they should; it's a test full of association and buzzwords, of course a program crushes it. That's literally nothing like real medicine.

2) AI in medicine will probably be more like chess engines in chess. For a few bucks, you can get a program for your laptop that can easily beat any human in chess, and that has been the case for probably 20+ years now. How has that translated into the game? Mostly for niche positional analysis. It's still human vs human.

3) As far as AI taking human jobs, this reminds me of a good med student quiz: "does the patient in front of you have Parkinson's disease, or do they not? You can run any tests or imaging studies you want." A trained human can answer this in a few minutes; even in 20 years I doubt AI will be able to.

Medicine will be affected by AI for sure. But for many reasons, it'll be one of the last fields to be affected.

To be fair, the exam results are from STEP 3 not STEP 1, which are significantly different. These are mostly third-order problems where you have to infer the diagnosis, know the standard of care, and then understand how the situation deviates from the first-line or know some fact about the first-line treatment.

 

[voxveritatisetlucis]

Bro all you post is stuff like this. I assume you're dropping out of medical school and putting your remaining tuition money into NVDA?


As for the OP, I have many thoughts, but will keep it to a few:
1) Med students and laypeople put massively more weight into "AI is killing it on Step 1" than they should; it's a test full of association and buzzwords, of course a program crushes it. That's literally nothing like real medicine.

2) AI in medicine will probably be more like chess engines in chess. For a few bucks, you can get a program for your laptop that can easily beat any human in chess, and that has been the case for probably 20+ years now. How has that translated into the game? Mostly for niche positional analysis. It's still human vs human.

3) As far as AI taking human jobs, this reminds me of a good med student quiz: "does the patient in front of you have Parkinson's disease, or do they not? You can run any tests or imaging studies you want." A trained human can answer this in a few minutes; even in 20 years I doubt AI will be able to.

Medicine will be affected by AI for sure. But for many reasons, it'll be one of the last fields to be affected.

There is no point in quitting because at least residency and fellowship is a guaranteed job for 7-10 years. By then, almost nobody will have a job outside of trades and surgeons so at least being a resident is better than nothing

Some of my friends in finance with solid degrees and work experience are having trouble getting a job. If I quit med school, I wouldn’t be able to find anything

 

[mrbreakfast]

To be fair, the exam results are from STEP 3 not STEP 1, which are significantly different. These are mostly third-order problems where you have to infer the diagnosis, know the standard of care, and then understand how the situation deviates from the first-line or know some fact about the first-line treatment.

Oh, you mean the medical common knowledge exam? Step 3 (the multiple-choice section, anyway) is not significantly different from step 1, other than being more focused on clinical applicability. It's still mainly pattern recognition; it's also regarded as much easier. Source: I have taken both exams.

The real question you should be asking is, "how can I use A.I to be a superhuman doctor". Do you realize how fast A.I can parse through literature and guidelines to help answer questions?

That's the thing. Doctors don't want something that can "parse through literature." We already have that (uptodate), and we already see the impacts of people (NPs or MDs) trying to treat conditions they don't know much about, based on general sources like that. A small but notable proportion of the cases I see are people with a condition that their NP/PCP or inpatient team identified, looked up how to treat, and tried treating it, causing more complications than good.

So when I go into chatgpt and ask "what is the preferred treatment for [condition X]," I can get a list of answers. But if I ask how to choose one treatment, or the evidence behind those recommendations, chatgpt refers me to pubmed. Now, perhaps in the future that will change, but I don't see AI as much more than a supplement for medical students (or for curious patients) at this time. Also consider that, for many, many clinical questions, there are no answers in literature. Not to mention dealing with patients who don't want to follow the standard of care.

I'm not trying to discourage you, and would be very interested to learn the medical student experience with AI in the coming years. But I think the disconnect with early-training medical students is that it is assumed challenges in medical treatment arise from a lack of readily-available knowledge - i.e., the answer is in front of us, we just need help seeing it - and not from the facts that the human body - and medications we give it - do not always behave in a predictable manner, patients do not provide accurate histories, lab tests can be misleading, and that medicine is truly more of an art than a science. You will likely not appreciate this until residency.

 

[SSerenity]

Oh, you mean the medical common knowledge exam? Step 3 (the multiple-choice section, anyway) is not significantly different from step 1, other than being more focused on clinical applicability. It's still mainly pattern recognition; it's also regarded as much easier. Source: I have taken both exams.


That's the thing. Doctors don't want something that can "parse through literature." We already have that (uptodate), and we already see the impacts of people (NPs or MDs) trying to treat conditions they don't know much about, based on general sources like that. A small but notable proportion of the cases I see are people with a condition that their NP/PCP or inpatient team identified, looked up how to treat, and tried treating it, causing more complications than good.

So when I go into chatgpt and ask "what is the preferred treatment for [condition X]," I can get a list of answers. But if I ask how to choose one treatment, or the evidence behind those recommendations, chatgpt refers me to pubmed. Now, perhaps in the future that will change, but I don't see AI as much more than a supplement for medical students (or for curious patients) at this time. Also consider that, for many, many clinical questions, there are no answers in literature. Not to mention dealing with patients who don't want to follow the standard of care.

I'm not trying to discourage you, and would be very interested to learn the medical student experience with AI in the coming years. But I think the disconnect with early-training medical students is that it is assumed challenges in medical treatment arise from a lack of readily-available knowledge - i.e., the answer is in front of us, we just need help seeing it - and not from the facts that the human body - and medications we give it - do not always behave in a predictable manner, patients do not provide accurate histories, lab tests can be misleading, and that medicine is truly more of an art than a science. You will likely not appreciate this until residency.

I think you are viewing "A.I" through the narrow scope of ChatGPT.

For instance, www.DrOracle.ai states "Ask Dr.Oracle any medical question. Get instant, precise & in-depth explanations directly from up-to-date guidelines & clinical research (with citations)."

These are systems that do use evidence based resources. To discount the value of tools like this is to discount the value of text books. These systems are essentially textbooks that are perpetually up to date, that you can converse with. You already do that in a sense (by using the search bar on UptoDate), but just extremely inefficiently.

 

[DOVinciRobot]

All non surgical specialties will likely be replaced within 10 years. Why pay a physician 400k per year if an AI can do most of it for 30 per month. If you’re in these fields, taking out margin loans or mortgages to buy NVDIA, Microsoft, Google, Meta is prudent. If you’re in surgery, Intuitive Surgical and Stryker may be good hedges in case AI assisted robotics progresses faster than anticipated.

If you think that’s too short of a timeline, the iPhone only came out like 17 years ago. In 20 years, AI will surpass humans in everything

Not really, no. AI will help physicians immensely, but it won’t replace them until after it has replaced entire other fields before it takes down physicians. The AI that would be required to completely replace physicians sure isn’t going to cost $30 a month….

I’m not sure we’ll ever have computers taking away surgery or procedural specialties. The physical reactions required for complications that can arise during procedures/surgery would require that we have literal cyborgs that are basically computer humans. At that point humanity as a whole is simply screwed (see: Terminator).

 

[SurfingDoctor]

No A.I. is gonna manually disimpact that 80 yo whose descending colon is the size of a watermelon... that one will always be for the medical student.

 

[Tenk]

All non surgical specialties will likely be replaced within 10 years. Why pay a physician 400k per year if an AI can do most of it for 30 per month. If you’re in these fields, taking out margin loans or mortgages to buy NVDIA, Microsoft, Google, Meta is prudent. If you’re in surgery, Intuitive Surgical and Stryker may be good hedges in case AI assisted robotics progresses faster than anticipated.

If you think that’s too short of a timeline, the iPhone only came out like 17 years ago. In 20 years, AI will surpass humans in everything

Until the AI encounters a Karen and decides to format itself in some weird AI seppuku.

 

[voxveritatisetlucis]

No A.I. is gonna manually disimpact that 80 yo whose descending colon is the size of a watermelon... that one will always be for the medical student.

True but couldn’t a nurse just do these things

 

[SurfingDoctor]

True but couldn’t a nurse just do these things

That's not what ChatGPT told me...

 

[AvsPearTree]

There is no point in quitting because at least residency and fellowship is a guaranteed job for 7-10 years. By then, almost nobody will have a job outside of trades and surgeons so at least being a resident is better than nothing

Some of my friends in finance with solid degrees and work experience are having trouble getting a job. If I quit med school, I wouldn’t be able to find anything

Dude what is this logic? constantly dooming about medicine and yet wanting to pursue it solely for job security for seven years until our AI overlords take over? Even as a resident, you'll be worked down to the bone for 80+ hours a week to earn less than minimum wage. Don't you think you could quit right now and find a job paying 60k where you wouldn't have to work as hard as you would in residency? Why not get a head start?

Move to Florida and start a power washing business, those guys rake it in with almost no overhead and AI robots won’t be doing that for at least a few more years.

Work for the gov, insanely good job security! USPS is always looking for drivers for example and they get amazing benefits and average salary is 65k.

 

[voxveritatisetlucis]

Dude what is this logic? constantly dooming about medicine and yet wanting to pursue it solely for job security for seven years until our AI overlords take over? Even as a resident, you'll be worked down to the bone for 80+ hours a week to earn less than minimum wage. Don't you think you could quit right now and find a job paying 60k where you wouldn't have to work as hard as you would in residency? Why not get a head start?

Move to Florida and start a power washing business, those guys rake it in with almost no overhead and AI robots won’t be doing that for at least a few more years.

Work for the gov, insanely good job security! USPS is always looking for drivers for example and they get amazing benefits and average salary is 65k.

I like medicine and don’t pay any tuition due to scholarship so I feel like this is still the best option for me, but it will vary. If you had to take out 500k in loans now especially with the SAVE plan revoked, would it be worth it?

 

[mrbreakfast]

I like medicine and don’t pay any tuition due to scholarship so I feel like this is still the best option for me, but it will vary. If you had to take out 500k in loans now especially with the SAVE plan revoked, would it be worth it?

There is no point in quitting because at least residency and fellowship is a guaranteed job for 7-10 years. By then, almost nobody will have a job outside of trades and surgeons so at least being a resident is better than nothing

If you truly have zero loans (i.e. significantly lowering the financial benefit of staying in medicine for PSLF), and if you truly believe that in 7-10 years no one outside of trades will have jobs, it is objectively demonstrable that you should leave medicine and start another career. PGY pay raises are set barely above inflation; if you're intelligent enough to get a tuition scholarship for med school, you could get a job at a supermarket and quickly rise to a managerial position, with much higher pay, and potentially better long-term employment prospects. If you want to stay in medicine, become a radiology tech, which has short school, much higher pay than resident salary, and cannot easily be replaced by AI.

If you don't do that, and stay in medicine, maybe take it easy on the doom and gloom.

 

[Dral]

No A.I. is gonna manually disimpact that 80 yo whose descending colon is the size of a watermelon... that one will always be for the medical student.

Haha, been that med student. It was my last day on service too. Jerk resident that had me do it.

 

[qwerty89]

All non surgical specialties will likely be replaced within 10 years. Why pay a physician 400k per year if an AI can do most of it for 30 per month. If you’re in these fields, taking out margin loans or mortgages to buy NVDIA, Microsoft, Google, Meta is prudent. If you’re in surgery, Intuitive Surgical and Stryker may be good hedges in case AI assisted robotics progresses faster than anticipated.

If you think that’s too short of a timeline, the iPhone only came out like 17 years ago. In 20 years, AI will surpass humans in everything

And the rate of change in that product has slowed to a crawl.

OG iPhone to iPhone 4 after 3 years? Huge change.
iPhone 12 to 15? Meh. ACTION BUTTON. TITANIUM.

 

[stay_foolish]

All non surgical specialties will likely be replaced within 10 years. Why pay a physician 400k per year if an AI can do most of it for 30 per month. If you’re in these fields, taking out margin loans or mortgages to buy NVDIA, Microsoft, Google, Meta is prudent. If you’re in surgery, Intuitive Surgical and Stryker may be good hedges in case AI assisted robotics progresses faster than anticipated.

If you think that’s too short of a timeline, the iPhone only came out like 17 years ago. In 20 years, AI will surpass humans in everything

The main issue is not whether AI will be capable. It will take longer time to take humans out of the loop. We'll have rules in place that will protect Physicians for long time to come. But just Physician service alone is like 600 B$ annually. So it might be very lucrative for Insurance companies and Tech companies to come after this. Also healthcare is one of the most important topic in political debate. At the end of the day every patient needs a human touch to heal.

 

[AnonymousDoctorGuyPerson]

All non surgical specialties will likely be replaced within 10 years. Why pay a physician 400k per year if an AI can do most of it for 30 per month. If you’re in these fields, taking out margin loans or mortgages to buy NVDIA, Microsoft, Google, Meta is prudent. If you’re in surgery, Intuitive Surgical and Stryker may be good hedges in case AI assisted robotics progresses faster than anticipated.

If you think that’s too short of a timeline, the iPhone only came out like 17 years ago. In 20 years, AI will surpass humans in everything

It's very clear where you are on the medicine curve.

Before vaulting into it medicine it is hard, unknowable, seemingly ungraspable. Then as you go through med school it becomes smaller, more algorithmic.

Then as you progress further you realize just how nuanced medicine is. There just isn't good data for a lot of what we do in medicine, and how we perform medicine. AI requires data, we are far from AI that can do the nuanced cognitive thinking we do utilizing what we do know, the data that does exist, and understanding how some data is stronger than others or may or may not directly apply to the specific patient in front of us. We also rely heavily on our personal experiences, our "clinical gestalt," and frankly also our human touch.

If AI were to hit that point many more dominos would fall before


You make the point of why not arm APPs with AI. The way AI currently exists and how it functions presently could ever only maximally operate at an APP level. Using known data, algorithmic. You can't combine them to get a physician, you just get two NPs working together.

 

[Cornfed101]

We’re still using meditech, which I’m pretty sure hasn’t been updated in the last 40 years. It will be a long time before AI replaces my job

 

[LucidSplash]

We’re still using meditech, which I’m pretty sure hasn’t been updated in the last 40 years. It will be a long time before AI replaces my job

There is a non-MSDOS version of Meditech, an “update.” But it reminds me distinctly of Windows 95, so it’s definitely still behind the other major inpatient systems.

 

[Cornfed101]

There is a non-MSDOS version of Meditech, an “update.” But it reminds me distinctly of Windows 95, so it’s definitely still behind the other major inpatient systems.

I’m on board if the alert system is clippy popping on screen saying “hey! are you sure you want to order haldol for this 85yo dementia patient??”