Is it weird using Chatgpt to rate at my App and tell me my chances

[FutureDOCTORR123456789]

---

Members do not see this ad.

It sounds stupid, but is it the worst idea? Especially if I tell it to be harsh and to not go easy?

 

[LizzyM]

Who trained the Chatgpt? What is the source of their knowledge about what makes a good application? Some of what they spew is complete garbage.

 

[Mr.Smile12]

Probably about as good/weird as ...

 

[Goro]

Or reading these:

 

[doctorendgame]

AI can get things wrong. That's why you need to have some expertise to call it out on things it got wrong on. I asked an AI to read my personal statement as an adcom who is super skeptical about any premeds pursuing medicine and it had a lot of criticisms but as I pointed out its flawed reasoning in each argument, its assessment of my personal statement got stronger. But these would erroneous assessments too.

 

[doctorendgame]

This is one of the dangers of AI. If you don't have any expertise, including in critiquing medical school essays, you won't know what to call it out on when it does get things wrong. IYKYK

To those who don't know, a demon threatened to destroy the entire world and Atem used a spell to seal that demon away using his name as the key. To prevent the demon from being freed, his successor and surviving subjects had all records of his name destroyed. The spell also sealed Atem's soul away, sacrificing his life and he erased his own memories after he sealed away with the demon.

 

[Mr.Smile12]

Included in Becoming a Student Doctor course

 

[MrM45]

I did it...it was decently accurate so far

 

[Mr.Smile12]

I did it...it was decently accurate so far

How many em dashes did you get??? 🙂

Again, we wish we knew the dataset. Someone got good lottery numbers by asking...

 

[polymerization]

There should be a custom GPT in the app store called Pre-Health Advisor by @Lee that I suspect is trained on advice given by adcoms here. I think it is a step up from just vanilla ChatGPT, though, honestly, just vanilla ChatGPT would have been a godsend when I was starting out in college.

 

[chilly_md]

This is one of the dangers of AI. If you don't have any expertise, including in critiquing medical school essays, you won't know what to call it out on when it does get things wrong. IYKYK

To those who don't know, a demon threatened to destroy the entire world and Atem used a spell to seal that demon away using his name as the key. To prevent the demon from being freed, his successor and surviving subjects had all records of his name destroyed. The spell also sealed Atem's soul away, sacrificing his life and he erased his own memories after he sealed away with the demon.

It can be inconsistent as well when asked the same question, even on things that should have easier to read sources online such as your example.

 

[doctorendgame]

It can be inconsistent as well when asked the same question, even on things that should have easier to read sources online such as your example.

Yeah, but the biggest danger with AI imo is if you have 0 expertise, you won't know what to call it out on. I assume a good chunk here never read or watched yugioh, so I figure it would be a good example. If you have no expertise, you won't know which is the correct answer when it's inconsistent.

 

[chilly_md]

Yeah, but the biggest danger with AI imo is if you have 0 expertise, you won't know what to call it out on. I assume a good chunk here never read or watched yugioh, so I figure it would be a good example. If you have no expertise, you won't know which is the correct answer when it's inconsistent.

Yeah, they tend to phrase it in a structured manner that shows it is confident in the answer even when it is far off. The yugioh example's equivalent counterpart would be telling applicants to take the GRE, get 150 hours of hospital volunteering and they'd be a fine candidate.

 

[doctorendgame]

OH! I completely forgot. I did exactly what you did with Perplexity's Deep Research function and the initial feedback was off. Then I pointed out the inconsistencies with its reasoning and this was the revised assessment. No change or edits happened with the personal statement. Just me asking it to reassess based on what it overlooked. So yeah, I highly advise against this approach. It wasn't a one shot reassessment. I had it reassess each of its criticisms individually and the assessment became progressively positive.

Given I only have 2 interviews so far, it seems a lot of schools don't share the AI's sentiment.

FYI, I also ran my personal statement by a couple public health professors to ensure I correctly differentiated public health from what I was trying to accomplish.

 

[Mr.Smile12]

OH! I completely forgot. I did exactly what you did with Perplexity's Deep Research function and the initial feedback was off. Then I pointed out the inconsistencies with its reasoning and this was the revised assessment. No change or edits happened with the personal statement. Just me asking it to reassess based on what it overlooked. So yeah, I highly advise against this approach. It wasn't a one shot reassessment. I had it reassess each of its criticisms individually and the assessment became progressively positive.

Also, given I only have 2 interviews so far, it seems a lot of schools don't share the AI's sentiment.
View attachment 411790
View attachment 411791

I don't know how many schools you applied to; it can be argued that AI can't be far off because you only had 2 interviews (which is an alternate premed interpretation... not a position I would take). But the second step you took is very important for many applicants to realize when I ask what one's purpose as a physician is. Too often applicants' reasoning for medicine sounds to me like they would be as happy with a different profession... (you are concerned about mental health... why not clinical psych), but the point is made that you can't take the first output from an AI bot at face value. You have to stand up for yourself.

 

[doctorendgame]

I don't know how many schools you applied to; it can be argued that AI can't be far off because you only had 2 interviews (which is an alternate premed interpretation... not a position I would take). But the second step you took is very important for many applicants to realize when I ask what one's purpose as a physician is. Too often applicants' reasoning for medicine sounds to me like they would be as happy with a different profession... (you are concerned about mental health... why not clinical psych), but the point is made that you can't take the first output from an AI bot at face value. You have to stand up for yourself.

40, give or take. Admittedly, I haven't been rejected by schools that have rejected a significant portion of their applicant pool yet like Rochester and UChicago and I also got a secondary from UCSF, so there's that so you may be right that its assessment (I presume the second) isn't far off.

 

[nobadwaves]

I highly advise that you fully utilize AI to write your primary, secondaries, and various letters of intent or update. Utilizing AI will likely increase the odds of your competition being accepted and you being fast forwarded to reapplication in 2027.

 

[polymerization]

OH! I completely forgot. I did exactly what you did with Perplexity's Deep Research function and the initial feedback was off. Then I pointed out the inconsistencies with its reasoning and this was the revised assessment. No change or edits happened with the personal statement. Just me asking it to reassess based on what it overlooked. So yeah, I highly advise against this approach. It wasn't a one shot reassessment. I had it reassess each of its criticisms individually and the assessment became progressively positive.

Given I only have 2 interviews so far, it seems a lot of schools don't share the AI's sentiment.

FYI, I also ran my personal statement by a couple public health professors to ensure I correctly differentiated public health from what I was trying to accomplish.

View attachment 411790
View attachment 411791

Yeah, I don't like Perplexity. If you're avoiding paid subscriptions, consider Gemini Pro (Google One AI is free for a year with an .edu e-mail). Even ChatGPT 5(.1?) mini is available to the public for free with rate limits.

If you are really reading for comprehension, even the second response is a whole lot of nothing. The model has to do rhetorical gymnastics to come up with the outcome you prompted.

For example, it's true that schools look for evidence of innovation... but just proposing a project is not inherently innovative. Given that clinical informatics is a known field with an entire fellowship program supporting it, I struggle to see how the practice of using big health data to support administrative/policy decisions is innovative. Presumably we have always done this manually, technology is just making it faster (and even then, we have to be considerate about the conclusions we make from this data, since we don't presently have a way of "checking the work" without doing it manually, which defeats the purpose). Further, considering neither you nor your mentor collected the data you are using to guide the project in the first place, to call it innovative crosses the line from overstatement to outright fabrication. I'm not poo-poo-ing CI, I had several CI projects...you just have to find a way rhetorically to make the connection.

The remaining points are redundant and generic. Every physician will have a "commitment to the underserved," to one degree or another, by mere coincidence. You could be seeing patients at the most expensive health system on the planet and it would not preclude the possibility of caring for someone who was once living in a precarious situation. Your basic science and future clinical education are designed to help you translate research into evidence-based practice. And everyone aims to tell a story and show reflective capacity through their essays.

I think you would know you're on a good track when the AI is providing receipts. If you're going to make those claims, fine—every school wants people who show these qualities—but they have to be both factually accurate and true to you to be valuable. It needs to be connecting the dots between experiences and reflections so that you are not spending your characters congratulating yourself (which has a tendency to go down like a lead balloon).

Ultimately AI is prompt based, so if you want the outcome to be "strongly recommend for interview," you can get it to hallucinate something that makes sense if we were just to take its word for it. The hard work is making sure the shoe actually fits, and that a human can put two and two together without having to be so on the nose about it. I hope that makes sense!

 

[Rachapkis]

Most organizations and even most people would benefit from employing a VP of Common Sense. At this point in the evoluation of AI, asking it to evaluate your application strikes me as an interesting experiment but AI's answer is not something that I would rely on. However, why not develop the best prompt you can, pose the it to AI, and let your VP of Common Sense evaluate the answer? Who knows? AI may get some things wrong but provide interesting insights on others. What's more, I disagree with some of the sentiments above regarding how useful AI can be in helping to draft a personal statement or other components of your application. Relying exclusively on AI would be a mistake. However, I have found that AI helps to organize and clarify my writing, cuts down on typos and other errors, and reduces the number of drafts that I need to produce something worthwhile--all of which makes me a bit sad--like a buggywhip maker at the dawn of the auto industry--because I have spent too much of my time learning how to write. (If you like this response, it was drafted without the assistance of AI. If you don't, I blame ChatGBT!)

 

[doctorendgame]

Yeah, I don't like Perplexity. If you're avoiding paid subscriptions, consider Gemini Pro (Google One AI is free for a year with an .edu e-mail). Even ChatGPT 5(.1?) mini is available to the public for free with rate limits.

If you are really reading for comprehension, even the second response is a whole lot of nothing. The model has to do rhetorical gymnastics to come up with the outcome you prompted.

For example, it's true that schools look for evidence of innovation... but just proposing a project is not inherently innovative. Given that clinical informatics is a known field with an entire fellowship program supporting it, I struggle to see how the practice of using big health data to support administrative/policy decisions is innovative. Presumably we have always done this manually, technology is just making it faster (and even then, we have to be considerate about the conclusions we make from this data, since we don't presently have a way of "checking the work" without doing it manually, which defeats the purpose). Further, considering neither you nor your mentor collected the data you are using to guide the project in the first place, to call it innovative crosses the line from overstatement to outright fabrication. I'm not poo-poo-ing CI, I had several CI projects...you just have to find a way rhetorically to make the connection.

The remaining points are redundant and generic. Every physician will have a "commitment to the underserved," to one degree or another, by mere coincidence. You could be seeing patients at the most expensive health system on the planet and it would not preclude the possibility of caring for someone who was once living in a precarious situation. Your basic science and future clinical education are designed to help you translate research into evidence-based practice. And everyone aims to tell a story and show reflective capacity through their essays.

I think you would know you're on a good track when the AI is providing receipts. If you're going to make those claims, fine—every school wants people who show these qualities—but they have to be both factually accurate and true to you to be valuable. It needs to be connecting the dots between experiences and reflections so that you are not spending your characters congratulating yourself (which has a tendency to go down like a lead balloon).

Ultimately AI is prompt based, so if you want the outcome to be "strongly recommend for interview," you can get it to hallucinate something that makes sense if we were just to take its word for it. The hard work is making sure the shoe actually fits, and that a human can put two and two together without having to be so on the nose about it. I hope that makes sense!

I never once said "underserved" in my personal statement actually. I did emphasize heavily on innovation and made sure to mention my mentor and how he's helping me so it comes across as grounded, not someone who's due for a rude awakening, given our last conversation (thanks for the benefit of the doubt by the way). But yeah, highly recommend not using AI.

 

[Mr.Smile12]

I also worry about echo chamber effects... you may be told information that you want to hear from AI...

 

[deleted1228948]

chat I worry about the state of brain rot

why would you use gpt to rate your app lmao
gpt is a glazer

 

[olofmeister]

chat I worry about the state of brain rot

why would you use gpt to rate your app lmao
gpt is a glazer

Your medical school application—shows a compelling blend—of strength, maturity and—commitment. Your clinical and—volunteer work show that you don't just participate—you engage and reflect—thoughtfully.

(whole time I have 50 clinical hours)

 

[geopigeon]

chat I worry about the state of brain rot

why would you use gpt to rate your app lmao
gpt is a glazer

I know someone doing this, they tell me its still early in the cycle
i dont think chat knows the date

 

[doctorendgame]

Your medical school application—shows a compelling blend—of strength, maturity and—commitment. Your clinical and—volunteer work show that you don't just participate—you engage and reflect—thoughtfully.

(whole time I have 50 clinical hours)

quality over quantity. 🤣

 

[anon2]

Yeah, but the biggest danger with AI imo is if you have 0 expertise, you won't know what to call it out on. I assume a good chunk here never read or watched yugioh, so I figure it would be a good example. If you have no expertise, you won't know which is the correct answer when it's inconsistent.

And this is why, as a ex-FAANG senior software engineer with a PhD and pubs with hundreds of citations in AI, I know enough to never touch a LLM for any useful work task.

Oh, just let it read through the web and give me a starting point? The work of critically analyzing its output to detect possible inaccuracies and fix them is vastly harder than doing the damn job myself, as any experienced line supervisor can tell you. The benefit of having a team (read: some AI bot) is scalability, but I can trivially research a topic sufficiently with Google scholar alone in 10 minutes.

If you need research skills, writing skills, etc., train them. A magic shortcut that jumps you to 80% accuracy, while actively impeding you from improving beyond that, is worse than useless when the professional standard is 95+%.

Yeah, they tend to phrase it in a structured manner that shows it is confident in the answer even when it is far off. The yugioh example's equivalent counterpart would be telling applicants to take the GRE, get 150 hours of hospital volunteering and they'd be a fine candidate.

Repeat after me: a LLM is designed to do exactly one thing. Predict the next word of human input based on data trawled through the internet, etc., violating as much copyright as they can get away with. That's it.

Absolutely nothing about meaning, sense, let alone accuracy*. It generates arbitrary amounts of text that is maximally similar (on average) to the human input it has seen. Which means (1) its grammar and prose are impeccable, and (2) any correct information it spews out is purely incidental based on the historical patterns matching.

Here's an RCT preprint: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

Coding is actually one of the best possible use cases for AI tooling and hence the most developed testbed: (1) well-defined problem, (2) easy to automatically evaluate arbitrary output to a quantitative standard.

*: My colleagues are of course trying. In my professional opinion, the community is lightyears away from getting it right. One popular framework is reinforcement learning, where you get humans to tell the model whether it's doing the right thing or not. Two words: remember Tay?

 

[polymerization]

(2) any correct information it spews out is purely incidental based on the historical patterns matching.

Most people use a search engine the same way and with the same expectations—we know the internet is not exactly fair and balanced on the whole. I think the bigger problem is that it speaks with an authoritative enough tone to convince 90% of people using the technology that it knows something it doesn't. Or that it can even really "know" anything. Or that "it" even is an "it." But, for an exponentially growing set of more casual use cases, LLMs do just fine.

AI definitely isn't what people say it is (I giggle when people say AGI is imminent)... but you cannot argue with everyday, low-stakes applications that are just genuinely helpful.

I think having an LLM read all your essays and reflect back the questions you should be answering in those essays (why medicine, why now, why not x, y, z) was useful. It was also useful to realize I could have also been making implicit points I didn't mean to make (but could have reasonably been interpreted that way).

We are very far from uploading a transcript and some basic information into an LLM and having it spit out the ideal medical school application. I don't think anyone here had plans to attempt that. It would make sense that PhD engineers are not at work doing some stylistic editing on some cute little essays about breaking an arm and realizing you now owe a duty and debt to society to become your one true destiny: an orthopod.

 

[anonymooseee]

I did this at the beginning of the cycle for funsies. I plugged in all of my app/stats/school list/everything, and asked it to be honest, be extremely critical like an adcom, and give me an optimistic, realistic, and worst-case prediction of the numbers of II and As I will get. Here's what it told me:

-Optimistic: 12-16 II; 6-9 A; a few T20s
-Realistic: 6-9 II; 3-4 A; state-school + 1-2 strong privates
-Worst-case: 2-4 II; 0-2 A; just state-school

Ig it's kind of accurate so far in terms of numbers? I also asked it to predict which specific schools I'll get IIs and As at, but that's a lot less accurate lol.

 

[doctorendgame]

And this is why, as a ex-FAANG senior software engineer with a PhD and pubs with hundreds of citations in AI, I know enough to never touch a LLM for any useful work task.

Oh, just let it read through the web and give me a starting point? The work of critically analyzing its output to detect possible inaccuracies and fix them is vastly harder than doing the damn job myself, as any experienced line supervisor can tell you. The benefit of having a team (read: some AI bot) is scalability, but I can trivially research a topic sufficiently with Google scholar alone in 10 minutes.

If you need research skills, writing skills, etc., train them. A magic shortcut that jumps you to 80% accuracy, while actively impeding you from improving beyond that, is worse than useless when the professional standard is 95+%.



Repeat after me: a LLM is designed to do exactly one thing. Predict the next word of human input based on data trawled through the internet, etc., violating as much copyright as they can get away with. That's it.

Absolutely nothing about meaning, sense, let alone accuracy*. It generates arbitrary amounts of text that is maximally similar (on average) to the human input it has seen. Which means (1) its grammar and prose are impeccable, and (2) any correct information it spews out is purely incidental based on the historical patterns matching.

Here's an RCT preprint: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

Coding is actually one of the best possible use cases for AI tooling and hence the most developed testbed: (1) well-defined problem, (2) easy to automatically evaluate arbitrary output to a quantitative standard.

*: My colleagues are of course trying. In my professional opinion, the community is lightyears away from getting it right. One popular framework is reinforcement learning, where you get humans to tell the model whether it's doing the right thing or not. Two words: remember Tay?

Perplexity is a search engine. It's useful for finding research papers. But I'm also working on a project that I can't talk too much about because I risk doxxing myself, but I'm working on it with a physician mentor and just had a massive breakthrough thanks to Perplexity, so it's not as bad as you think it is.

 

[anon2]

Most people use a search engine the same way and with the same expectations—we know the internet is not exactly fair and balanced on the whole. I think the bigger problem is that it speaks with an authoritative enough tone to convince 90% of people using the technology that it knows something it doesn't. Or that it can even really "know" anything. Or that "it" even is an "it." But, for an exponentially growing set of more casual use cases, LLMs do just fine.

AI definitely isn't what people say it is (I giggle when people say AGI is imminent)... but you cannot argue with everyday, low-stakes applications that are just genuinely helpful.

I think having an LLM read all your essays and reflect back the questions you should be answering in those essays (why medicine, why now, why not x, y, z) was useful. It was also useful to realize I could have also been making implicit points I didn't mean to make (but could have reasonably been interpreted that way).

We are very far from uploading a transcript and some basic information into an LLM and having it spit out the ideal medical school application. I don't think anyone here had plans to attempt that. It would make sense that PhD engineers are not at work doing some stylistic editing on some cute little essays about breaking an arm and realizing you now owe a duty and debt to society to become your one true destiny: an orthopod.

The critical area which makes all the difference: a search engine is a retrieval tool. It shows you links to pages, where you get to scrutinize the primary source and make your own decisions.

A LLM shows you a constructed synthesis where the connection to the (possibly nonexistent, ffs) sources are uninterpretable, and critical thinking is effectively impossible without already knowing the correct answer, whence it becomes a pure efficiency tool in the shape of a foot-autocannon.

AI, in its current massive ANN form, does have an actual use: in very high sensitivity rule-out screening tests, to reduce the workload in vigilance tasks which humans are evolutionarily terrible at. Think audits.

Perplexity is a search engine. It's useful for finding research papers. But I'm also working on a project that I can't talk too much about because I risk doxxing myself, but I'm working on it with a physician mentor and just had a massive breakthrough thanks to Perplexity, so it's not as bad as you think it is.

My question is this: was any component of the "AI" part of the search engine at all necessary for your breakthrough? How different would your experience have been had you simply used Pubmed? This includes the skills that you would have otherwise trained in the absence of the AI crutch.

As I posted originally, all these startup AI tools exist because they violate copyright on a planetary scale as their fundamental business model. Consider your ethical sensibilities.

A special purpose tool will always do vastly better than a general one: no free lunch. LLM engineering has already moved into specializing many, many different use cases, as it is in fact the standard way to obtain acceptable performance.

At this point AI (NLP) serves mostly as a "conversational" front-end interface to the specific tool, and the actual search itself doesn't actually need AI to begin with. This is the approach of WolframAlpha, which predates the AI malignancy.

 

[doctorendgame]

My question is this: was any component of the "AI" part of the search engine at all necessary for your breakthrough? How different would your experience have been had you simply used Pubmed? This includes the skills that you would have otherwise trained in the absence of the AI crutch.

As I posted originally, all these startup AI tools exist because they violate copyright on a planetary scale as their fundamental business model. Consider your ethical sensibilities.

A special purpose tool will always do vastly better than a general one: no free lunch. LLM engineering has already moved into specializing many, many different use cases, as it is in fact the standard way to obtain acceptable performance.

At this point AI (NLP) serves mostly as a "conversational" front-end interface to the specific tool, and the actual search itself doesn't actually need AI to begin with. This is the approach of WolframAlpha, which predates the AI malignancy.

It was more than just finding a simple peer reviewed journal and yes, it was borderline necessary for my breakthrough.

Fortunately, AI for science and medicine vs art is like night and day.

 

[polymerization]

The critical area which makes all the difference: a search engine is a retrieval tool. It shows you links to pages, where you get to scrutinize the primary source and make your own decisions.

A LLM shows you a constructed synthesis where the connection to the (possibly nonexistent, ffs) sources are uninterpretable, and critical thinking is effectively impossible without already knowing the correct answer, whence it becomes a pure efficiency tool in the shape of a foot-autocannon.

AI, in its current massive ANN form, does have an actual use: in very high sensitivity rule-out screening tests, to reduce the workload in vigilance tasks which humans are evolutionarily terrible at. Think audits.

If you notice I'm actually not debating that point... I'm just saying the internet more broadly also doesn't exactly cite its sources, so nothing's really changed. You'd have to already be an expert to do expert-level research.

When I was growing up in the age of dial-up, the popular thing to say was "Wikipedia is not a legitimate source." Now they say "LLMs make mistakes." If anything, it is the "telephone game" effect where synthesis of already unreliable information is likely to stray even further from the truth, but it doesn't seem to be as much a problem of categorical lack of utility as it is one of epistemology and how accessible ground truth data could realistically ever be, assuming current limitations.

I will reiterate there are more proximal use cases that are much more tolerant even of wrong answers. Sometimes you need a sounding board to bounce concepts off of. I've found it immensely helpful in identifying literature that might help me further explore some specific topic of interest.

And again, I'll reiterate I am just as skeptical, but some folks really push the hypothetical limit of what these technologies are actually being used for. If people are categorically using LLMs to do important, consequential work, they should be held responsible in accordance to whatever integrity policy governs their work products... I suspect that a retraction of LLMs (if even feasible at this point) would just result in the contraction of use to precisely the high-discretion knowledge workers privileged enough to access them, which makes your specific point moot. Lazy SWEs will still attempt to vibe-code.

The paternalistic attitude just kinda tickles me, especially in the setting of the populist "personal responsibility" FAFO politics of 2025. Pretty soon full-grown adults are going to need accountability buddies to cross the streets. People are already unaliving themselves from chatting with a computer program. Personally, I think the genie is already out of the bottle. Darwin and Freud win again.

 

[JakAttk]

I did this at the beginning of the cycle out of curiosity, so I'll contribute the results here in the interest of science. I asked it to be very conservative and sent my full primary and secondaries.

Summary – Conservative Prediction
Total predicted IIs: 15
Total predicted Acceptances: 12 (including 1 from WL movement)
Total predicted Waitlists: 3 (1 likely to accept, 2 likely to reject)
Total predicted Post-II rejections: 7
Total predicted Pre-II rejections: 14

I'm currently sitting at 7 II and 1 A. I think like many prediction tools, Chat overvalues stats. Some of its data points are going to be 20 year old forum posts when stats mattered more and admissions were less competitive.

 

[anon2]

If you notice I'm actually not debating that point... I'm just saying the internet more broadly also doesn't exactly cite its sources, so nothing's really changed. You'd have to already be an expert to do expert-level research.

When I was growing up in the age of dial-up, the popular thing to say was "Wikipedia is not a legitimate source." Now they say "LLMs make mistakes." If anything, it is the "telephone game" effect where synthesis of already unreliable information is likely to stray even further from the truth, but it doesn't seem to be as much a problem of categorical lack of utility as it is one of epistemology and how accessible ground truth data could realistically ever be, assuming current limitations.

I will reiterate there are more proximal use cases that are much more tolerant even of wrong answers. Sometimes you need a sounding board to bounce concepts off of. I've found it immensely helpful in identifying literature that might help me further explore some specific topic of interest.

And again, I'll reiterate I am just as skeptical, but some folks really push the hypothetical limit of what these technologies are actually being used for. If people are categorically using LLMs to do important, consequential work, they should be held responsible in accordance to whatever integrity policy governs their work products... I suspect that a retraction of LLMs (if even feasible at this point) would just result in the contraction of use to precisely the high-discretion knowledge workers privileged enough to access them, which makes your specific point moot. Lazy SWEs will still attempt to vibe-code.

The paternalistic attitude just kinda tickles me, especially in the setting of the populist "personal responsibility" FAFO politics of 2025. Pretty soon full-grown adults are going to need accountability buddies to cross the streets. People are already unaliving themselves from chatting with a computer program. Personally, I think the genie is already out of the bottle. Darwin and Freud win again.

I agree with most of this. Mind, I wrote my dissertation on explainability of AI, and am headed towards the regulatory part of the space, mostly to prevent the current malignancy from metastasizing even worse.

On a practicality standpoint, AI detection tools aren't super great, especially when there are people who grossly mistrain their writing skills on chatbot output, and basically pattern themselves into emulating one. The clear ethical requirement breaks down when enforcement is extremely difficult, while the offending side can spam with ~0 effort. This is the same losing war as with disinformation, seen clearly a few years ago.

FYI, sounding boards don't necessarily need to even respond to you at all. See rubber ducking, which is a legitimate method that I (and colleagues) use. As to finding relevant literature, my PhD training included the ability to reverse-engineer key search terms based on scanning tangents. I grew up in the same age of dial-up. Again, AI advancement may be useful, but it is not necessary. Time will tell whether radiologists become extinct (they won't).

In my opinion, the moral hazard of LLMs is as follows:

(1) massive ungated availability to the public
(2) output that is, by design, on its face extremely persuasive yet grossly inadequate for entrustable tasks
(3) a deliberate commercial hype train touting it for everything, with absurd malinvestment
(4) including dangerously inappropriate applications such as robot "psychotherapy"
(5) general public unawareness, particularly other subject experts, of the very sharp but nonobvious limitations of LLM*
(6) training is based on a business model that is non-viable without massive stealing from all content creators
(7) while using arbitrarily high computing resources (~10x normal search)

I don't advocate the removal of LLMs, which is censorship and as you said, paternalism. What I do professionally is the usual education efforts combined with regulatory advocacy.

*: It's not even the issues I've mentioned so far. Machine learning doesn't solve arbitrarily hard pattern matching problems, it transforms them into, in some aspects, the even harder problem of data cleaning, hidden bias elimination, and essentially trying to force the infinitely literal computer into doing what a human thinks is the "correct" thing. At a scale where manual checking is ~impossible.

 

[polymerization]

*: It's not even the issues I've mentioned so far. Machine learning doesn't solve arbitrarily hard pattern matching problems, it transforms them into, in some aspects, the even harder problem of data cleaning, hidden bias elimination, and essentially trying to force the infinitely literal computer into doing what a human thinks is the "correct" thing. At a scale where manual checking is ~impossible.

I agree most strongly with this in particular, which aligns well with what I said earlier about this applied AI study.

I struggle to see how the practice of using big health data to support administrative/policy decisions is innovative. Presumably we have always done this manually, technology is just making it faster (and even then, we have to be considerate about the conclusions we make from this data, since we don't presently have a way of "checking the work" without doing it manually, which defeats the purpose).

I was on a computer vision project that gave me a totally different view of what I was doing. It seemed like every time the model struggled discerning two similar datapoints, the answer would be to project the data to a higher level of abstraction ad infinitum until it could tell them apart. It was a CNN, so I mean, duh, that's how it works, but it was my first time building from an empty Jupyter notebook.

It felt like we weren't actually encoding intelligence so much as it felt like playing Battleship with code, just making endless guesses and trying to get the output to be useful more so by coincidence than any genuine content knowledge. And that's the trade off with supervised learning right? The more you hold the model's hand, the more we can explain its output, even if it's garbage... but what we're seeing more and more often are really convincing, seemingly correct responses that we cannot possibly interrogate once we are working with self-supervised models like LLMs.

I agree with the moral hazards you present, though my guess is that safety/superalignment will take a backseat. The internet has changed a lot in the last year. Almost everything I see is AI slop, even from trusted, official sources. The government itself is producing misinformation to such a flagrant degree that sometimes, I wish I were the one hallucinating. Even ChatGPT "knows" Tylenol doesn't cause autism, and yet, that is what the HHS secretary is shilling. And this is just one situation in which the truth is stranger than fiction, with seemingly just as deleterious effects on the broader population.

In other words, AI is only one (and by no means the most dangerous) source of questionable information.

I loved what @Rachapkis said. Life's tough, get a helmet. Everyone needs a VP of Common Sense.

For anyone interested in what we're talking about, I loved this video by Dr. Angela Collier, astrophysicist/science-communicator-extraordinaire: