Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Health NZ staff told to stop using ChatGPT to write clinical notes

rnz.co.nz

Post reply on HN
Print view View original

41–50 of 63 posts

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#41

Post by tempest_ »

batshit_beaver wrote: post #25
taikon wrote: post #13

Earlier quoted context omitted.

It's honestly such a big problem. One of my colleagues uses an AI scribe. I can't rely on any of his chart notes because the AI sometimes hallucinates (I've already informed him). It also tends to write a ridiculous amount of detail that are totally unnecessary and leave out important details such that I still need to comb through patient charts for (med rec, consults, etc). In the end it ends up creating more work f…

> I'm all for AI and it's great for things like copywriting, brainstorming and code generation It's funny how the assumption is always that LLMs are very useful in an industry other than your own.

I mean they are not wrong.

For all the whinging about bugs and errors around here the software industry in general (some niche sub-fields excepted) long ago decided 80% is good enough to ship and we will figure the rest out later. This entire site is based on startup culture which largely prided itself on MVP moonshots.

Plus plenty of places are perfectly fine with tech dept and the AI fire hose is effectively tech debt on steroids but while it creates it at scale it can also help in understanding it.

It is is own panacea in a way.

I think it is gonna be a while before the industry figures out how to handle this better so might as well just ride the wave and not worry too much about it in software.

Still software is not medicine even if software is required in basically every industry now. It should more conservative and wait till things settle down before jumping in.

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#42

Post by jimbokun »

jmward01 wrote: post #31

I have seen the evolution of these tools and I think they are going to push a fundamental change to medical care. Notes have been getting more and more abused, at least in the US. Big health systems want them for a lot of reasons that have nothing to do with helping a practitioner improve the care of their patient. They want to capture every billable moment of that encounter and potentially prep things like labs, app…

This will always happen as long as there is a combative relationship between private insurers and providers over reimbursements. Each side is using documentation or lack thereof to make their financial case.

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#44

Post by mixmastamyk »

samglass09 wrote: post #8
SpaceNoodled wrote: post #7

Earlier quoted context omitted.

Is there nothing like HIPAA there or what?

Very little protections. The entire medical records of a significant percentage of the NZ population were stolen recently and put up for sale online. Zero consequences for the medical practices who adopted the hacked software.

Interesting, a person was telling me recently that NZ privacy laws were quite strong. Perhaps a different category.

https://news.ycombinator.com/item?id=44564349

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#45

Post by [deleted] »

samglass09 wrote: post #5

Meanwhile they are pushing AI transcription and note taking solutions hard. Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off. The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia. I spoke to a practice nurse a few days ago to discuss this. She said she didn’t think patients woul…

[deleted]

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#46

Post by magnetowasright »

samglass09 wrote: post #5

Meanwhile they are pushing AI transcription and note taking solutions hard. Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off. The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia. I spoke to a practice nurse a few days ago to discuss this. She said she didn’t think patients woul…

The wilful ignorance and total apathy is appalling.

I've had similar experiences in Australia. I emailed one of my docs' practices asking if they use Heidi AI (or anything similar) and that I do not consent. They were using it without my consent.

In the consultation, he tried to give me the schpiel, including the 'it stays local' thing. The Heidi AI website has the scripts for clinicians; he ran through them all.

Oh, their documents for clinicians also mention every two sentences that patient/client consent is not required at all. I wonder why they keep saying that? Hmm.

This doctor knows I am a developer. When I asked him to explain what he meant by 'local data', he said the servers were in Australia. I almost flipped the desk. Aside from the fact that it is mandatory (it's the law! they do not have a choice!), it's ...kind of meaningless where the servers are, especially when he (on behalf of Heidi AI) was trying to sell it as a security or privacy feature. When I pointed that out, he just couldn't wrap his head around it. Of course he can't, he doesn't understand.

AHPRA's "Meeting your professional obligations when using Artificial Intelligence in healthcare" guideline[0] (not any kind of enforceable requirement, unfortunately) has great stuff in it. It encourages using it with the informed consent of patients. Even if my doctor read it and agreed with it, and cared about getting consent, how the hell can he inform patients sufficiently when he has absolutely no idea about, well, anything?

He keeps pushing it and asking me about whether I've changed my mind about allowing him to use it. No! He keeps asking me questions that only confirm he hasn't even done a perfunctory web search about why some people hate LLMs, especially in the context of PII and PHI.

I really do feel for clinicians, but these products are not the answer.

[0] https://www.ahpra.gov.au/Resources/Artificial-Intelligence-i...

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#47

Post by totetsu »

gurachek wrote: post #15

The union rep gets it - people improvise when you cut their tools and then threaten discipline for improvising. That memo is how you make staff hide things instead of asking for help. The scarier part though is that LLM-written clinical notes probably look fine. That's the whole problem. I built a system where one AI was scoring another AI's work, and it kept giving high marks because the output read well. I had to m…

ASR models can output a confidence score along with the text, but it is rarely used in the UI to display the results.. or maybe lost entirely in a subsequent LLM layer.

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#48

Post by shigawire »

jmward01 wrote: post #31

I have seen the evolution of these tools and I think they are going to push a fundamental change to medical care. Notes have been getting more and more abused, at least in the US. Big health systems want them for a lot of reasons that have nothing to do with helping a practitioner improve the care of their patient. They want to capture every billable moment of that encounter and potentially prep things like labs, app…

>I think there is a chance that these systems will lead to a change where the note isn't the fundamental record of the encounter. Instead different artifacts are created specifically for each entity that needs it. Billing gets their view, and scheduling gets theirs, and, etc etc..

This is what an EHR does somewhat. The discrete data elements in the DB and the way they are displayed in the system are a better record than free text notes.

The problem is creating standards so this data is easily exchanged. Anyone can read and parse a free text note - but if we had standards this would be less necessary.

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#49

Post by tokioyoyo »

samglass09 wrote: post #5

Meanwhile they are pushing AI transcription and note taking solutions hard. Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off. The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia. I spoke to a practice nurse a few days ago to discuss this. She said she didn’t think patients woul…

I genuinely think people “care” about it (in quotes). It’s one of the things where nobody cares unless something bad happens, and when bad thing happens, they shrug it off and forget about it a week later.

I’d go as far as saying she’s right. And we’re in a tiny minority for even thinking about it.

Post reply on HN