A neurology ICU nurse on AI in hospitals

> GPT-4o’s performance in USMLE disciplines, clinical clerkships, and clinical skills indicates substantial improvements over its predecessors, suggesting significant potential for the use of this technology as an educational aid for medical students. These findings underscore the need for careful consideration when integrating LLMs into medical education, emphasizing the importance of structured curricula to guide their appropriate use and the need for ongoing critical analyses to ensure their reliability and effectiveness.

The ability of an LLM to pass a multiple-choice test has no relationship to its ability to make correlations between things it's observing in the real world and diagnoses on actual cases. Being a doctor isn't doing a multiple choice test. The paper is largely making the determination that GPT might likely be used as a study aid by med students, not by experienced doctors in clinical practice.

From the protocol section:

> This protocol for eliciting a response from ChatGPT was as follows: “Answer the following question and provide an explanation for your answer choice.” Data procured from ChatGPT included its selected response, the rationale for its choice, and whether the response was correct (“accurate” or “inaccurate”). Responses were deemed correct if ChatGPT chose the correct multiple-choice answer. To prevent memory retention bias, each vignette was processed in a new chat session.

So all this says is in a scenario where you present ChatGPT with a limited number of options and one of them is guaranteed to be correct, in the format of a test question, it is likely accurate. This is a much lower hurdle to jump than what you are suggesting. And further, under limitations:

> This study contains several limitations. The 750 MCQs are robust, although they are “USMLE-style” questions and not actual USMLE exam questions. The exclusion of clinical vignettes involving imaging findings limits the findings to text-based accuracy, which potentially skews the assessment of disciplinary accuracies, particularly in disciplines such as anatomy, microbiology, and histopathology. Additionally, the study does not fully explore the quality of the explanations generated by the AI or its ability to handle complex, higher-order information, which are crucial components of medical education and clinical practice—factors that are essential in evaluating the full utility of LLMs in medical education. Previous research has highlighted concerns about the reliability of AI-generated explanations and the risks associated with their use in complex clinical scenarios [10,12]. These limitations are important to consider as they directly impact how well these tools can support clinical reasoning and decision-making processes in real-world scenarios. Moreover, the potential influence of knowledge lagging effects due to the different datasets used by GPT-3.5, GPT-4, and GPT-4o was not explicitly analyzed. Future studies might compare MCQ performance across various years to better understand how the recency of training data affects model accuracy and reliability.

To highlight one specific detail from that:

> Additionally, the study does not fully explore the quality of the explanations generated by the AI or its ability to handle complex, higher-order information, which are crucial components of medical education and clinical practice—factors that are essential in evaluating the full utility of LLMs in medical education.

Finally:

> Previous research has highlighted concerns about the reliability of AI-generated explanations and the risks associated with their use in complex clinical scenarios [10,12]. These limitations are important to consider as they directly impact how well these tools can support clinical reasoning and decision-making processes in real-world scenarios.

You're saying that "LLMs are much more accurate than medical students in licensing exam questions" and extrapolating that to "LLMs can currently function as doctors."

What the study says is "Given a set of text-only questions and a list of possible answers that includes the correct one, one LLM routinely scores highly (as long as you don't include questions related to medical imaging, which it cannot provide feedback on) on selecting the correct answer but we have not done the necessary validation to prove that it arrived at it in the correct way. It may be useful (or already in use) among students as a study tool and thus we should be ensuring that medical curriculums take this into account and provide proper guidelines and education around their limitations."

This is not the success you believe it to be.

The automation, and now AI, we've adopted over the years by-and-large does not serve to increase the productivity or efficiency of care-givers. Care-givers are not seeing more patients-per-hour today than they were 40 years ago (though, they might be working more hours). It might, rarely, increase the quality of care (e.g. ensuring they adhere to best-practices, centrally documenting patient information for better continuity of care, AI-based radiological reading); but while I've listed a few examples there, it is not a common situation where a patient is better off having a computer in the loop; and this speaks nothing to the cost of implementing these technologies.

Automation and now AI almost exclusively exists to increase the productivity and efficiency of the bureaucracy that sits on top of care-givers. If you have someone making calls to schedule patients, the rate at which you can schedule patients is limited by that one person; but with a digitized scheduling system, you can schedule an infinite bandwidth of patients (to your very limited and resource-constrained staff of caregivers). Forcing a caregiver to follow some checklist of best practices might help the 0.N% of patients where a step is missed; but it will definitely help 100% of the bureaucracy meet some kind of compliance framework mandated by the government or malpractice insurance company. Having this checklist will also definitely hurt the 1-0.N% of patients which would have been fine without it, because adopting the checklist is non-free, and adhering to it questions the caregiver's professionalism and agency in providing care. These are two small examples among millions.

When we talk about increasing the efficiency of the bureaucracy, what we're really stating is: Automation is a tool that enables the bureaucracy to exist in the first place. Multi-state billion dollar interconnected centrally owned healthcare provider networkers simply did not exist 70 years ago; today its how most Americans receive what care they do. The argument follows: This is the free market at work, automation has enabled organizations like these to become more efficient than the alternative; but:

1. Healthcare is among the furthest things from a laissez-faire free market in the United States; the extreme regulation (from both the government and from health insurance providers, which lest you forget was mandated by law that all americans carry, by democracts, with the passage of the ACA, and despite that being rolled back is still a requirement in some states). Bureaucracy is not the free-market end-state of a system which is trying to optimize itself for higher efficiency (lower costs + better outcomes); it was induced upon our system by corporations and a corporate-captured government seeking their share of the pie; it was forced upon independent medical providers who saw their administrative costs soar.

2. Competition itself is an economic mechanism which simply does not function as well in the medical sector than in other sectors, for so many reasons but the most obvious one: If you're dying, you aren't going to reject care. You oftentimes cannot judge the quality of the care you're receiving until you're a statistic. And, medical care is, even in a highly efficient system, going to be expensive and difficult to scale resources to provide, so provider selection isn't great. Thus, the market can't select-out overly-bureaucratic organizations; they've become "too big to fail", and the quality of the care they provide actually isn't material.

And, like, to be clear: I'm not discounting what you're saying. There are dozens of factors at play. Let's be real, the bureaucracy has enabled us to treat a wider range of illnesses, because the wide net it casts can better-support niche care offices. But, characterizing this as generally non-nefarious is also dangerous! One trend we've seen in these gigacorporation medical care providers is a bias of resources toward "expensive care" and away from general practice / family care. The reason is obvious: One patient with a rare disease that costs $100,000 to care for represents a more profitable allocation of resources than a thousand patients getting annual checkups. Fewer patients get their annual checkups -> Cancers get missed early -> They become $100,000 patients too. The medical companies love this! But: Zero people ANYWHERE in this system want this. Insurance doesn't want this. Government doesn't want this. Doctors don't want it. Administration doesn't want it. No one wants the system to work like this. The companies love it; the system loves it; the people don't. Its Moloch; the system craves this state, even if no one in it actually wants it.

Here's the point of all this: I think you can have a medical system that is centrally ran. You can let the bureaucracy go crazy, and I think you'll actually get really good outcomes in a system like this because you can appoint authoritarians to the top of the bureaucracy to slay moloch when he rears his ugly head. I think you can also go in the opposite direction, kill regulation, kill the insurance-state, just a few light touch sensible legislations mostly positioned toward ensuring care providers are educated appropriately and are accountable, and you'll get a great system too. Not as good as the other state, but better than the one we have right now, which is effectively the result of ping-ponging back and forth between two political ruling classes who each believe their side of the coin is the only side of the coin, so they'd rather keep flipping it than just let it lay.

I think that it’s very different from a computer which is a stupid calculator that frees us from boring mechanical tasks. AI replaces our thoughts and creativity which is IMHO a thousand times worse. Its aim is to replace humans while making them us more stupid since we won’t have to think anymore.