I know at least one GP that has stopped using Heidi Health for transcription. He (and as I've noticed with transcriptions from my medical professionals) has noticed many errors, far too many to be comfortable. Things might improve, but not yet.
Health NZ staff told to stop using ChatGPT to write clinical notes
21–30 of 63 posts
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#22Earlier quoted context omitted.
The New Zealand Chief Digital Officer allowed Australian cloud providers to be used as there weren't suitable NZ data centers and this was many years ago.
Health NZ adopted Snowflake. It was about costs/fancy tech. We have always had data centres. Nobody *needs* snowflake. They could have used Apache Spark.
[deleted]
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#23FYI, AI adoption in health in NZ is moving forward, for example https://www.rnz.co.nz/news/national/589774/emergency-doctors... This is just about not using free/public AI tools.
that's mentioned in OP article.
Heidi is frustratingly consistent at hallucinating stuff. I've seen it in almost all of the dozen or so summaries I've had from medical people recently (surgeon, physio, consultant). A GP I know tried for a month and then was like 'it's not worth the risk exposure to me or my patients'.
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#24The union rep gets it - people improvise when you cut their tools and then threaten discipline for improvising. That memo is how you make staff hide things instead of asking for help. The scarier part though is that LLM-written clinical notes probably look fine. That's the whole problem. I built a system where one AI was scoring another AI's work, and it kept giving high marks because the output read well. I had to m…
Physicians need to have it pounded into them that every hallucination is downstream harm. AI has no place in medicine. If they insist on it, then all transcripts must be stored with the raw audio. Which should be accessible side by side, with lines of transcript time coded. It's the only way to actually use these safely, while guarding against hallucinations.
Raw audio is a cool idea! I've seen a similar approach in other domains, "keep the source of truth accessible so you can verify the AI output against it".
I wouldn't go as far as "no place in medicine" though. The Heidi scribe tool mentioned in the article is a good example, because in the end it's the doctor who reviews and signs off.
IMO the problem is AI doing the work with no human verification step, but I can 100% agree I don't want to have vibe-doctor for my next surgery/consult :D
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#25Meanwhile they are pushing AI transcription and note taking solutions hard. Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off. The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia. I spoke to a practice nurse a few days ago to discuss this. She said she didn’t think patients woul…
It's honestly such a big problem. One of my colleagues uses an AI scribe. I can't rely on any of his chart notes because the AI sometimes hallucinates (I've already informed him). It also tends to write a ridiculous amount of detail that are totally unnecessary and leave out important details such that I still need to comb through patient charts for (med rec, consults, etc). In the end it ends up creating more work f…
> I'm all for AI and it's great for things like copywriting, brainstorming and code generation
It's funny how the assumption is always that LLMs are very useful in an industry other than your own.
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#26Meanwhile they are pushing AI transcription and note taking solutions hard. Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off. The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia. I spoke to a practice nurse a few days ago to discuss this. She said she didn’t think patients woul…
It's honestly such a big problem. One of my colleagues uses an AI scribe. I can't rely on any of his chart notes because the AI sometimes hallucinates (I've already informed him). It also tends to write a ridiculous amount of detail that are totally unnecessary and leave out important details such that I still need to comb through patient charts for (med rec, consults, etc). In the end it ends up creating more work f…
> I'm all for AI and it's great for things like copywriting, brainstorming and code generation
That's funny. I would have said the same thing about your field prior to reading your comment.
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#27Meanwhile they are pushing AI transcription and note taking solutions hard. Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off. The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia. I spoke to a practice nurse a few days ago to discuss this. She said she didn’t think patients woul…
Didn't Health NZ just suffer a major data breach and have patient records ransomed?
There were two serious breaches recently but they were at private companies not HNZ.
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#28Meanwhile they are pushing AI transcription and note taking solutions hard. Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off. The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia. I spoke to a practice nurse a few days ago to discuss this. She said she didn’t think patients woul…
It's honestly such a big problem. One of my colleagues uses an AI scribe. I can't rely on any of his chart notes because the AI sometimes hallucinates (I've already informed him). It also tends to write a ridiculous amount of detail that are totally unnecessary and leave out important details such that I still need to comb through patient charts for (med rec, consults, etc). In the end it ends up creating more work f…
My (extensive) experience with LLM code generation is that it has the same issues you describe in your field. Hallucinations, over-engineering, misses important requirements/patterns.
But engineers have these same problems. The key is that the content creator (engineers for codegen, doctors for medicine) is still responsible for the output of the AI, as if they wrote it themselves. If they make a mistake with an AI (eg, include false data - hallucinations), they should be held accountable in the same way they would if they made a mistake without it.
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#29I know at least one GP that has stopped using Heidi Health for transcription. He (and as I've noticed with transcriptions from my medical professionals) has noticed many errors, far too many to be comfortable. Things might improve, but not yet.
It's Gell-Mann amnesia: you notice the errors in fields where you're an expert.
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#30I know at least one GP that has stopped using Heidi Health for transcription. He (and as I've noticed with transcriptions from my medical professionals) has noticed many errors, far too many to be comfortable. Things might improve, but not yet.
This is where I'm at as a GP. Every few months I give Heidi another try, but I haven't noticed any real improvement over the last two years. It spends lots of words on trivial nonsense and misses clinically significant points and sometimes entire issues. It takes far more time to review and fix the notes than it saves in typing. Presumably it will be good enough one day, but it's not there yet.