Hold on. Does this mean ER diagnoses are marginally better than pure chance?
No, because randomly guessing from a list of diagnoses is not 50/50
Good point.
OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors
491–500 of 500 posts
Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors
#492Earlier quoted context omitted.
That’s not really an argument, it is central to my point. The current system does exhibit those issues and it is by human creativity and outliers that we have some points of escape from it. Codifying and distilling it removes the points of escape.
Sure, it's not an argument if you ignore half of what I wrote. Your previous reply to me was similarly disconnected from the comment it was responding to. Were the systemic issue to remain unaddressed AI would certainly be expected to make it worse. But it could be addressed if there was the will to do so. In fact AI could actually be leveraged to improve things by taking on the role that physicians play now thus fre…
In the same way AI frees up SREs and developers to focus on all the downtime and bugfixes they’ve caused by using AI now?
Humans don’t get to drop down to 2 9’s though… we just die.
Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors
#493Earlier quoted context omitted.
>It's having a general understanding/view of the "baseline", aka healthy anatomy. This is something LLMs will never have You're making the mistake of conflating AI with LLMs. I don't think LLMs will reliably be better than a board of doctors. But an Expert System probably will (if it isn't already). That's literally what they were created for. The biggest downside of LLMs IMO isn't the millions of Jules wasted on tra…
The article is about an LLM, the GGP explicitly cites LLMs. It's not the GP conflating them.
The comment thread was about AI, not LLM.
Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors
#494Earlier quoted context omitted.
I'm not. I understand the difference and also that through improvements to the core models as well as harnesses, LLMs are able to handle an increasing share of tasks. I also understand that these things will continue to improve until AI can automate entire jobs. You, on the other hand, are confusing LLMs from the past with current SOTA LLMs, which can tell how many rs are in strawberry.
It's tech bros like you that are to blame for the shortage of radiologists supporting statements since 2016 which state that radiologists will disappear. Your great SOTA LLMs will tell you to walk 5 mins to the car wash instead of taking the car.
Nope, wrong again - SOTA LLMs get the car wash thing right.
If you're going to have strong opinions on a topic, it would behoove you to keep up with it.
Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors
#495Earlier quoted context omitted.
Doctors make errors all the time though, so the real argument is about the error percentage. If AIs is lower then it's safer (but it's hard to have that convo, I recognise). Besides; this article was about diagnosis not prescribing. It's pretty obvious, I think, that diagnosis is one area where AI will perform extremely well in the long run. I think there are two metrics; the first is outright misdiagnosis, which stu…
The bar for making ai useful is much lower though. It's enough to be better than nothing. Large populations also in the technically rich countries simply do not have access to a doctor. in Poland which has a free public Healthcare it takes literal years to get a single appointment sometimes.
I agree with you, but seperate point in many respects - the conversation was about replacing existing robust medical infrastructure.
I fully agree that AI could extend access; but to build on what others have said too, lack of physical diagnostics is an issue as is the lack of physical tech infrastructure.
Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors
#496Earlier quoted context omitted.
I would love to hear of any specific example, I will happily either show you how to order it or learn something myself.
Arterial blood gas. Calcium score (may not count as a lab). Skin biopsy for cancer (does it count as a lab?). I'm unaware how to order my own troponin if I think I've had a heart attack (not that that's one I should DIY diagnosis). Prostate specific antigen.
Several of those are more procedures than labs. Of course you can't get someone to do a procedure on you for free. Arterial sticks and biopsies may have nontrivial risks (and commensurate liability risk for the performing provider).
PSA and troponin seem trivial to get. Did you look?
Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors
#497Earlier quoted context omitted.
In healthcare, HIPAA/GDPR equivalent would block this. Let's be realistic in our discussion; this is not the same as google buying up a library worth of books, scanning and destroying them
There are other countries, and the patients in them all have similar data
Similar data?!
We have wildly heterogeneous data just within the US!
And again, how exactly is this interface going to work? How does the AI determine how hard to press on an abdomen, and where, and how does it press there once it has that information?
Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors
#498Earlier quoted context omitted.
I agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors),…
To answer your question: talking to a human. Medicine is so much more than "knowledge, experience, and pattern matching", as any patient ever can attest to. Why is it so hard for some people to understand that humans need other humans and human problems can't be solved with technology?
>Why is it so hard for some people to understand that humans need other humans and human problems can't be solved with technology?
Perhaps because they have been persistently failed by other humans, but not by technology; and/or because they believe they don't have the right and/or capability to improve other humans, but may be able to improve the technology?
For complex human reasons, I am effectively deprived of healthcare, in a country where healthcare is socialized. Other services provided by the state are also gatekept by my nominal healthcare provider, making certain "normal" things, indeed things that are required of me in order to participate in society, technically impossible.
(I will not go into concrete detail, because when I request help from anyone with my situation, the "help" consists of implorations to comply with painful nonsense, combined with a random helping of rudeness and idiocy. Yes, we exist.)
If I had "open source" access to just the "knowledge, experience, and pattern matching" that is presumably still involved in medical practice besides this nebulous "talking to a human" (which the other human usually actively works to make impossible, having been fundamentally socialized into language use by means of violence - and thus, the more intelligent my interlocutor, the more quickly they begin to feel threatened by understanding what I'm talking about), I would be able to maintain my body to a better standard than the standard of care that the institutional medical establishment has kindly deigned to make available to me.
Meanwhile, I don't even have "open source" access to the designs of the motor vehicle which hauls said body around. Being a car mechanic is so much more than "knowledge, experience, and pattern matching", too - it's being a part of a web of trust and tacit collusion; a.k.a. a guild.
Come to think of it, I don't even have "open source" access to the internals of the device I'm writing this from. Though at least in the domain of computing there are valiant attempts to produce libre software and hardware.
I'm happy that "talking to a human" has helped you. In my case, requesting help has a >10% likelihood to endanger my life, >25% to imperil my health, and >50% to degrade my sanity. Rough ballpark, eh?
A fully "open source" OS, device, vehicle, body, and mind, would presumably allow me to solve my immediate problems without needing other humans to perform the inexpicably painful sacrifice of comprehending my communications. Without access to these basics, consent to "healthcare" is impossible; we just put our lives in the hands of the medical technopriesthood and hope they don't leave us with permanently crossed fingers.
Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors
#499AI beating doctors on text-based diagnosis isn't surprising. LLMs are pattern-matching machines trained on millions of medical cases. But the real test is the physical exam. Can AI tell the difference between someone faking pain and someone with a ruptured appendix? Between a panic attack and a heart attack? That's what doctors do that AI can't.
Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors
#500The x-ray study where AI beat radiologists without access to x-rays is hilarious and terrifying. It means the benchmark was broken, not the AI. We're going to see a lot of "AI beats humans" studies that just mean the humans were asked to do something unnatural. Real medicine isn't a multiple-choice test.