Towards accurate differential diagnosis with large language models • Hacker News Board

Re: Towards accurate differential diagnosis with large language models

#31

Post by dogmatism » Tue, Dec 05, 2023, 2:16 AM UTC

arcticfox wrote: post #29

dogmatism wrote: post #27

Earlier quoted context omitted.

Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit and for a follow up visit, read the prior visit's note and orders, search/track down all the test results (may be at more than one institution with multiple EHR s…

> Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit LLMs were born to do this. > or even just be able to take verbal orders like it used to be: "get a CBC, CMP, TSH, and CXR" GPT-4 of course knew what that meant,…

I know an LLM can do this when given the data in an ingestible form

it's the taking of data from disparate sources including literal paper faxes that's the hard part

Re: Towards accurate differential diagnosis with large language models

#32

Post by dogmatism » Tue, Dec 05, 2023, 2:23 AM UTC

modeless wrote: post #30

dogmatism wrote: post #27

Earlier quoted context omitted.

Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit and for a follow up visit, read the prior visit's note and orders, search/track down all the test results (may be at more than one institution with multiple EHR s…

I mean, have you used GPT-4? Free form text is its bread and butter, and yes, it can absolutely ingest images and read text in them, and combine that with other information from multiple sources in structured or unstructured text or image form, no problem! What you are describing sounded decades away last year. Today it's not something you can rely on quite yet if you need accuracy. But if you still believe it's deca…

see my comment above

Pt sees Dr in office A, has labs and scans ordered. Labs order is printed out and given to pt, who takes it to lab A. Scan #1 is ordered, and order faxed to facility B, who independently calls, schedules, performs, and interprets scan #1, same for scan #2, but at a different facility. Office A has no electronic link to any of these facilities

How does LLM get the data?

A human knows the patterns in the area, and calls around for it, then collates it and feeds in to office A's EHR, where if lucky, LLM can do it's thing

LLM is only saving the bare minimum of the work

In an industry that is still reliant on the fax, I'm not holding my breath for the advent of the LLM savior

Re: Towards accurate differential diagnosis with large language models

#33

Post by modeless » Tue, Dec 05, 2023, 2:33 AM UTC

dogmatism wrote: post #32

modeless wrote: post #30

Earlier quoted context omitted.

I mean, have you used GPT-4? Free form text is its bread and butter, and yes, it can absolutely ingest images and read text in them, and combine that with other information from multiple sources in structured or unstructured text or image form, no problem! What you are describing sounded decades away last year. Today it's not something you can rely on quite yet if you need accuracy. But if you still believe it's deca…

see my comment above Pt sees Dr in office A, has labs and scans ordered. Labs order is printed out and given to pt, who takes it to lab A. Scan #1 is ordered, and order faxed to facility B, who independently calls, schedules, performs, and interprets scan #1, same for scan #2, but at a different facility. Office A has no electronic link to any of these facilities How does LLM get the data? A human knows the patterns…

In the situation you describe humans fail constantly. My experience with the medical system has decidedly not been that magically competent back office people call every office you've ever visited to assemble your complete medical history from scraps and effectively organize it for the doctor. It's a constant game of telephone where you have to do most of the work yourself, calling around and giving instructions to receptionists to give to other people to get this stuff done. They can't even communicate reliably between the nurses and doctors in the same damn office to avoid asking the same questions multiple times every time you visit.

What you describe, if it worked reliably, would be an advance over current human-level average performance. And AI will absolutely be able to do it, too. Probably within ten years, though it will take longer to deploy after the capabilities are there.

Re: Towards accurate differential diagnosis with large language models

#34

Post by Sai_ » Tue, Dec 05, 2023, 2:49 AM UTC

dogmatism wrote: post #31

arcticfox wrote: post #29

Earlier quoted context omitted.

> Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit LLMs were born to do this. > or even just be able to take verbal orders like it used to be: "get a CBC, CMP, TSH, and CXR" GPT-4 of course knew what that meant,…

I know an LLM can do this when given the data in an ingestible form it's the taking of data from disparate sources including literal paper faxes that's the hard part

You’re being dogmatic about your, what seems to me, objections to LLM skills.

Making data ingestible is just laborious and unappetising to an engineer. So hand it over to a data entry operator who’ll do this happily for a wage.

I’m not sure why the issue of making data available to an LLM is a failure of the LLM.

Btw, OpenAI’s LLM can very well parse images, read handwriting, fix sloppy spellings, and extract objects from unstructured data.

Re: Towards accurate differential diagnosis with large language models

#35

Post by corethree » Tue, Dec 05, 2023, 2:54 AM UTC

doix wrote: post #26

This doesn't really surprise me given my experience with doctors. If it's a common problem then doctors do fine. As soon as you're not the common case or have something somewhat rare, it becomes a bit of a crap shoot, at least in the UK when you deal primarily with your GP (general practitioner). I have a friend with Crohn's who was feeling low energy. I was a gym bro at the time and convinced him to take a testoster…

[flagged]

Re: Towards accurate differential diagnosis with large language models

#36

Post by [deleted] » Tue, Dec 05, 2023, 2:58 AM UTC

westurner wrote: post #3

Differential diagnosis > Machine differential diagnosis: https://en.wikipedia.org/wiki/Differential_diagnosis CDSS: Clinical Decision Support System: https://en.wikipedia.org/wiki/Clinical_decision_support_syst... Treatment decision support: https://en.wikipedia.org/wiki/Treatment_decision_support : > Treatment decision support consists of the tools and processes used to enhance medical patients’ healthcare decision-…

[deleted]

Re: Towards accurate differential diagnosis with large language models

#37

Post by comradesmith » Tue, Dec 05, 2023, 3:56 AM UTC

dogmatism wrote: post #32

modeless wrote: post #30

Earlier quoted context omitted.

I mean, have you used GPT-4? Free form text is its bread and butter, and yes, it can absolutely ingest images and read text in them, and combine that with other information from multiple sources in structured or unstructured text or image form, no problem! What you are describing sounded decades away last year. Today it's not something you can rely on quite yet if you need accuracy. But if you still believe it's deca…

see my comment above Pt sees Dr in office A, has labs and scans ordered. Labs order is printed out and given to pt, who takes it to lab A. Scan #1 is ordered, and order faxed to facility B, who independently calls, schedules, performs, and interprets scan #1, same for scan #2, but at a different facility. Office A has no electronic link to any of these facilities How does LLM get the data? A human knows the patterns…

Not hard to strap GPT voice to a phone line, and provide it with some information of local labs and their contact information.

Also you said the office doesn’t have an electronic link but your example workflow includes the phone ;)

Re: Towards accurate differential diagnosis with large language models

#38

Post by vidarh » Tue, Dec 05, 2023, 3:59 AM UTC

doctorpangloss wrote: post #9

resters wrote: post #5

If it's not already obvious, LLMs are going to be doing most of the mental work currently performed by doctors, lawyers, accountants, etc. I have already nearly stopped using Google search for anything, in favor of GPT-4. GPT-4 has helped me very quickly prototype things that I normally would have had to spend hours researching. GPT-4 has also created custom curriculum for me to help me learn various things for which…

> If it's not already obvious, LLMs are going to be doing most of the mental work currently performed by doctors, lawyers, accountants, etc. While I don't know you personally and I'm not talking about you specifically, "just" feeling very excited about something doesn't Make This About You, the prognostications don't actually help you Be a Part of This. This is the same energy as being really into COVID-19 (what was…

My GP starts with phone or video consultations 100% of the time. There is no option to see them in person without a referral after a phone or video consult. (I chose them because of the video focus)

Will a good LLM based "consultation" fairly often have to refer you to a face to face consultation? Probably, but of my consultations over the last few years, a significant majority were resolved purely remotely.

Re: Towards accurate differential diagnosis with large language models

#39

Post by whatshisface » Tue, Dec 05, 2023, 4:02 AM UTC

lumost wrote: post #15

doctorpangloss wrote: post #9

Earlier quoted context omitted.

> If it's not already obvious, LLMs are going to be doing most of the mental work currently performed by doctors, lawyers, accountants, etc. While I don't know you personally and I'm not talking about you specifically, "just" feeling very excited about something doesn't Make This About You, the prognostications don't actually help you Be a Part of This. This is the same energy as being really into COVID-19 (what was…

I strongly doubt that the LLM club will remain exclusive for very long, or that the innovation in this space was restricted to 10k people. At a minimum, I’ve already begun using gpt-4 as a consultant for my fathers medical issues. I can reasonably get confirmation/dis confirmation of whether a medical experts opinion is BS or not. As anyone who has dealt with the medical system in the US can attest - the majority of…

This adds a whole new dimension to AI safety, it's one thing to dumb it down so it can't do bad stuff, but when you do that you're making it give worse medical advice to people who are going to take that advice.

Re: Towards accurate differential diagnosis with large language models

#40

Post by vidarh » Tue, Dec 05, 2023, 4:03 AM UTC

dogmatism wrote: post #31

arcticfox wrote: post #29

Earlier quoted context omitted.

> Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit LLMs were born to do this. > or even just be able to take verbal orders like it used to be: "get a CBC, CMP, TSH, and CXR" GPT-4 of course knew what that meant,…

I know an LLM can do this when given the data in an ingestible form it's the taking of data from disparate sources including literal paper faxes that's the hard part

There's no need for paper faxes - fax is a digital format. But of course, if you intentionally and pointlessly output to formats that make input to an LLM more work, it will be hard.

Other than that dealing with disparate data formats is something LLMs can already do very well, and that can be improved fairly significantly by putting a "straightjacket" on it (making it use APIs or selecting only tokens that fits a given grammar)