Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Towards accurate differential diagnosis with large language models

arxiv.org

Post reply on HN
Print view View original

21–30 of 99 posts

Re: Towards accurate differential diagnosis with large language models

#21

Post by 0xDEAFBEAD »

fnordpiglet wrote: post #8
guender wrote: post #2

Yikes, do I read this correctly and the LLM alone outperforms clinician + LLM?

This has come up before in similar contexts. Model based decision decisions tend to do better than clinicians alone or clinicians with the model based decision system in general. Despite this it’s been effectively impossible to deploy such systems in a clinical setting due to many issues but one is that clinicians aren’t willing to cede their ground and patients aren’t willing to believe the machine over a human.

Is there any way for me to get ahold of one of these models for my own private use?

Re: Towards accurate differential diagnosis with large language models

#22

Post by rickysahu »

We are doing similar work at GenHealth.ai and getting sota results on some evals (not yet published). Our approach is very different from LLMs in that we are using a medical coding vocabulary and we are training transformers on actual patient histories. We have an API if anyone here wants to build on it. Oh and we are hiring

Re: Towards accurate differential diagnosis with large language models

#23

Post by rafaelero »

fnordpiglet wrote: post #8
guender wrote: post #2

Yikes, do I read this correctly and the LLM alone outperforms clinician + LLM?

This has come up before in similar contexts. Model based decision decisions tend to do better than clinicians alone or clinicians with the model based decision system in general. Despite this it’s been effectively impossible to deploy such systems in a clinical setting due to many issues but one is that clinicians aren’t willing to cede their ground and patients aren’t willing to believe the machine over a human.

I don't see how those models couldn't be used to at least offer a second opinion. They are pretty cheap to run. I doubt patients would complain about that.

Re: Towards accurate differential diagnosis with large language models

#25

Post by kelseyfrog »

guender wrote: post #2

Yikes, do I read this correctly and the LLM alone outperforms clinician + LLM?

Is that across all subspecialties?

I hear internists talking about +LL and -LL all the time and I'd hope that reflected some higher degree of rationality when reasoning out diagnoses, but perhaps not?

Re: Towards accurate differential diagnosis with large language models

#26

Post by doix »

This doesn't really surprise me given my experience with doctors. If it's a common problem then doctors do fine. As soon as you're not the common case or have something somewhat rare, it becomes a bit of a crap shoot, at least in the UK when you deal primarily with your GP (general practitioner).

I have a friend with Crohn's who was feeling low energy. I was a gym bro at the time and convinced him to take a testosterone test (because all problems re caused by low T when you're a gym bro).

His doctor wouldn't even entertain the idea, saying he's a young man and it's very unlikely that he'd have low T. He did the test privately and his T is significantly below normal. If you Google, there are actually many papers showing correlation between Crohn's and low T. I bet an AI would find it.

Similarly, doctors missed my mum's recent cancer diagnosis. She also had factors that would make her more susceptible to breast cancer, googling finds many papers that show causation.

The problem is that those things aren't extremely common and haven't made their way to NICE guidelines or whatever GPs use.

Not that I'm blaming doctors, they have 10 minute appointments and don't have time to do anything. I'm sure AI would recommend significantly more lab tests which would put even more pressure on the NHS.

Re: Towards accurate differential diagnosis with large language models

#27

Post by dogmatism »

resters wrote: post #5

If it's not already obvious, LLMs are going to be doing most of the mental work currently performed by doctors, lawyers, accountants, etc. I have already nearly stopped using Google search for anything, in favor of GPT-4. GPT-4 has helped me very quickly prototype things that I normally would have had to spend hours researching. GPT-4 has also created custom curriculum for me to help me learn various things for which…

Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit

and for a follow up visit, read the prior visit's note and orders, search/track down all the test results (may be at more than one institution with multiple EHR systems that don't interchange) and summarize

or even just be able to take verbal orders like it used to be: "get a CBC, CMP, TSH, and CXR"

The NEJM CPC's are after the hard work has already been done to get the data to present the case

Re: Towards accurate differential diagnosis with large language models

#29

Post by arcticfox »

dogmatism wrote: post #27
resters wrote: post #5

If it's not already obvious, LLMs are going to be doing most of the mental work currently performed by doctors, lawyers, accountants, etc. I have already nearly stopped using Google search for anything, in favor of GPT-4. GPT-4 has helped me very quickly prototype things that I normally would have had to spend hours researching. GPT-4 has also created custom curriculum for me to help me learn various things for which…

Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit and for a follow up visit, read the prior visit's note and orders, search/track down all the test results (may be at more than one institution with multiple EHR s…

> Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit

LLMs were born to do this.

> or even just be able to take verbal orders like it used to be: "get a CBC, CMP, TSH, and CXR"

GPT-4 of course knew what that meant, although I didn't. I told it to roleplay a potential follow-up:

> "Patient with fatigue, weight loss, mild fever. CBC shows anemia, elevated WBC. CMP reveals elevated liver enzymes. TSH normal. CXR clear. Consider ESR, CRP, ANA, and abdominal ultrasound. Possible infection or autoimmune condition. Referral to a hematologist may be warranted."

Re: Towards accurate differential diagnosis with large language models

#30

Post by modeless »

dogmatism wrote: post #27
resters wrote: post #5

If it's not already obvious, LLMs are going to be doing most of the mental work currently performed by doctors, lawyers, accountants, etc. I have already nearly stopped using Google search for anything, in favor of GPT-4. GPT-4 has helped me very quickly prototype things that I normally would have had to spend hours researching. GPT-4 has also created custom curriculum for me to help me learn various things for which…

Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit and for a follow up visit, read the prior visit's note and orders, search/track down all the test results (may be at more than one institution with multiple EHR s…

I mean, have you used GPT-4? Free form text is its bread and butter, and yes, it can absolutely ingest images and read text in them, and combine that with other information from multiple sources in structured or unstructured text or image form, no problem!

What you are describing sounded decades away last year. Today it's not something you can rely on quite yet if you need accuracy. But if you still believe it's decades away then you need to update your perception of what's going on in AI right now. Forget the free GPT-3.5 or Bard, you really need to try GPT-4 to understand what's about to happen.

Post reply on HN