Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Health NZ staff told to stop using ChatGPT to write clinical notes

rnz.co.nz

Post reply on HN
Print view View original

11–20 of 63 posts

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#11

Post by simmerup »

javascriptfan69 wrote: post #4
burnte wrote: post #2

Yeah, no privacy or security there. There are some tools explicitly designed at helping healthcare providers produce better notes faster, and a couple of them are AMAZING. I'm an AI-half-empty guy, I'm keenly aware of its shortcomings and deploy it thoughtfully, and even with my skepticism there are a couple of tools that are just plain great. I think using LLMs to create overviews and summaries is a great use of the…

The one my doctor was using got my obs numbers completely wrong. We had to correct them at the end of the consultation.

Gotta break a few eggs to save 2 minutes of thinking and work

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#12

Post by Ucalegon »

lights0123 wrote: post #9
SpaceNoodled wrote: post #7

Earlier quoted context omitted.

Is there nothing like HIPAA there or what?

Many AI companies, including Azure with their OpenAI hosting, are more than willing to sign privacy agreements that allow processing sensitive medical data with their models.

The devil is in the details. For example, OAI does not have regional processing for AU [0] and their ZDR does not cover files[1]. Anthropic's ZDR [2] also does not cover files, so you really need to be careful, as a patient/consumer, to ensure that your health, or other sensitive data, that is being processed by SaaS frontier models is not contained in files. Which is asking a a lot of the medical provider to know how their systems work, they won't, which is why I will never opt in.

[0] https://developers.openai.com/api/docs/guides/your-data#whic...

[1] https://developers.openai.com/api/docs/guides/your-data#stor...

[2] https://platform.claude.com/docs/en/build-with-claude/zero-d...

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#13

Post by taikon »

samglass09 wrote: post #5

Meanwhile they are pushing AI transcription and note taking solutions hard. Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off. The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia. I spoke to a practice nurse a few days ago to discuss this. She said she didn’t think patients woul…

It's honestly such a big problem. One of my colleagues uses an AI scribe. I can't rely on any of his chart notes because the AI sometimes hallucinates (I've already informed him). It also tends to write a ridiculous amount of detail that are totally unnecessary and leave out important details such that I still need to comb through patient charts for (med rec, consults, etc). In the end it ends up creating more work for me. And if my colleague ever gets a college complaint I have no clue how he's gonna navigate any AI generated errors. I'm all for AI and it's great for things like copywriting, brainstorming and code generation. But from what I'm seeing, it's creating a lot more headache in the clinical setting.

If you're why doesn't this guy just check the AI scribe notes? Well, probably because with the amount of detail it writes, he'd be better off writing a quick soap note.

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#14

Post by samglass09 »

jimjimjim wrote: post #10
samglass09 wrote: post #5

Meanwhile they are pushing AI transcription and note taking solutions hard. Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off. The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia. I spoke to a practice nurse a few days ago to discuss this. She said she didn’t think patients woul…

The New Zealand Chief Digital Officer allowed Australian cloud providers to be used as there weren't suitable NZ data centers and this was many years ago.

Health NZ adopted Snowflake. It was about costs/fancy tech. We have always had data centres. Nobody *needs* snowflake. They could have used Apache Spark.

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#15

Post by gurachek »

The union rep gets it - people improvise when you cut their tools and then threaten discipline for improvising.

That memo is how you make staff hide things instead of asking for help.

The scarier part though is that LLM-written clinical notes probably look fine. That's the whole problem. I built a system where one AI was scoring another AI's work, and it kept giving high marks because the output read well. I had to make the scorer blind to the original coaching text before it started catching real issues. Now imagine that "reads well, isn't right" failure mode in clinical documentation.

Nobody's re-reading the phrasing until a patient outcome goes wrong.

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#17

Post by rconti »

taikon wrote: post #13
samglass09 wrote: post #5

Meanwhile they are pushing AI transcription and note taking solutions hard. Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off. The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia. I spoke to a practice nurse a few days ago to discuss this. She said she didn’t think patients woul…

It's honestly such a big problem. One of my colleagues uses an AI scribe. I can't rely on any of his chart notes because the AI sometimes hallucinates (I've already informed him). It also tends to write a ridiculous amount of detail that are totally unnecessary and leave out important details such that I still need to comb through patient charts for (med rec, consults, etc). In the end it ends up creating more work f…

It feels very much like AI is creating AI lock-in (if not AI _vendor_ lock-in) by creating so much detailed information that it's futile to consume it without AI tools.

I was updating some gitlab pipelines and some simple testing scripts and it created 3 separate 300+ line README type metadata files (I think even the QUCIKSTART.md was 300 lines).

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#18

Post by salawat »

gurachek wrote: post #15

The union rep gets it - people improvise when you cut their tools and then threaten discipline for improvising. That memo is how you make staff hide things instead of asking for help. The scarier part though is that LLM-written clinical notes probably look fine. That's the whole problem. I built a system where one AI was scoring another AI's work, and it kept giving high marks because the output read well. I had to m…

Physicians need to have it pounded into them that every hallucination is downstream harm. AI has no place in medicine. If they insist on it, then all transcripts must be stored with the raw audio. Which should be accessible side by side, with lines of transcript time coded. It's the only way to actually use these safely, while guarding against hallucinations.

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#19

Post by fhub »

samglass09 wrote: post #5

Meanwhile they are pushing AI transcription and note taking solutions hard. Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off. The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia. I spoke to a practice nurse a few days ago to discuss this. She said she didn’t think patients woul…

I spec'd up an implementation of this that uses a hardware button with colors that is in reach of either party. The customer went with a different vendor based on price/"complexity"/training.

Re: Health NZ staff told to stop using ChatGPT to write clinical notes

#20

Post by tjpnz »

samglass09 wrote: post #5

Meanwhile they are pushing AI transcription and note taking solutions hard. Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off. The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia. I spoke to a practice nurse a few days ago to discuss this. She said she didn’t think patients woul…

Didn't Health NZ just suffer a major data breach and have patient records ransomed?
Post reply on HN