I think there is a chance that these systems will lead to a change where the note isn't the fundamental record of the encounter. Instead different artifacts are created specifically for each entity that needs it. Billing gets their view, and scheduling gets theirs, and, etc etc... It will, hopefully, give the practitioners a chance to get back to focusing on the patient and not ensuring their note quality captured one more billable code. Of course the negative is also likely to happen here too. As practitioners spend less time on the note they will likely not get that back in time with individual patients, but instead on seeing more patients. It will also likely lead to higher bills as the health systems do start squeezing more out of every encounter. There is no perfect here when profit is the driving motivator but with this much change happening I can only hope that it causes the industry as a whole to shake up enough to maybe find a new better optimum to land in.
Health NZ staff told to stop using ChatGPT to write clinical notes
31–40 of 63 posts
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#32Earlier quoted context omitted.
It's honestly such a big problem. One of my colleagues uses an AI scribe. I can't rely on any of his chart notes because the AI sometimes hallucinates (I've already informed him). It also tends to write a ridiculous amount of detail that are totally unnecessary and leave out important details such that I still need to comb through patient charts for (med rec, consults, etc). In the end it ends up creating more work f…
My (extensive) experience with LLM code generation is that it has the same issues you describe in your field. Hallucinations, over-engineering, misses important requirements/patterns. But engineers have these same problems. The key is that the content creator (engineers for codegen, doctors for medicine) is still responsible for the output of the AI, as if they wrote it themselves. If they make a mistake with an AI (…
Are you willing to put your life in the hands of these people fully using the machines to do everything?
Acting like that smart people aren't getting one shot'ed by these machines is very dangerous. Even worse is how quickly your skills actual degrade. If knew my doctor was using anything LLM related, I would switch doctors.
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#33Earlier quoted context omitted.
Many AI companies, including Azure with their OpenAI hosting, are more than willing to sign privacy agreements that allow processing sensitive medical data with their models.
The devil is in the details. For example, OAI does not have regional processing for AU [0] and their ZDR does not cover files[1]. Anthropic's ZDR [2] also does not cover files, so you really need to be careful, as a patient/consumer, to ensure that your health, or other sensitive data, that is being processed by SaaS frontier models is not contained in files. Which is asking a a lot of the medical provider to know ho…
The models are licensed to Microsoft, and you pay them for the inference.
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#34Earlier quoted context omitted.
The devil is in the details. For example, OAI does not have regional processing for AU [0] and their ZDR does not cover files[1]. Anthropic's ZDR [2] also does not cover files, so you really need to be careful, as a patient/consumer, to ensure that your health, or other sensitive data, that is being processed by SaaS frontier models is not contained in files. Which is asking a a lot of the medical provider to know ho…
Azure OpenAI is not the same as paying OpenAI directly. While you may not be able to pay OpenAI for them to run models in Australia, you can pay Azure: https://azure.microsoft.com/en-au/pricing/details/azure-open... The models are licensed to Microsoft, and you pay them for the inference.
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#35Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#36Meanwhile they are pushing AI transcription and note taking solutions hard. Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off. The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia. I spoke to a practice nurse a few days ago to discuss this. She said she didn’t think patients woul…
The New Zealand Chief Digital Officer allowed Australian cloud providers to be used as there weren't suitable NZ data centers and this was many years ago.
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#37The union rep gets it - people improvise when you cut their tools and then threaten discipline for improvising. That memo is how you make staff hide things instead of asking for help. The scarier part though is that LLM-written clinical notes probably look fine. That's the whole problem. I built a system where one AI was scoring another AI's work, and it kept giving high marks because the output read well. I had to m…
Physicians need to have it pounded into them that every hallucination is downstream harm. AI has no place in medicine. If they insist on it, then all transcripts must be stored with the raw audio. Which should be accessible side by side, with lines of transcript time coded. It's the only way to actually use these safely, while guarding against hallucinations.
I think any person using 'AI' knows it makes mistakes. In a medical note, there are often errors at present. A consumer of a medical note has to decide what makes sense and what to ignore, and AI isn't meaningfully changing that. If something matters, it's asked again in follow up.
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#38AI doesn't forget and soon all of new zealanders will have their health histories internalised by AI so it can individually calculate insurance premiums without knowing why....
In fact, it's human transcribers who chose whether to forget the details of a case or whether to share the details of an especially funny patient with their buddies at the bar.
Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#39Re: Health NZ staff told to stop using ChatGPT to write clinical notes
#40Meanwhile they are pushing AI transcription and note taking solutions hard. Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off. The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia. I spoke to a practice nurse a few days ago to discuss this. She said she didn’t think patients woul…
It's honestly such a big problem. One of my colleagues uses an AI scribe. I can't rely on any of his chart notes because the AI sometimes hallucinates (I've already informed him). It also tends to write a ridiculous amount of detail that are totally unnecessary and leave out important details such that I still need to comb through patient charts for (med rec, consults, etc). In the end it ends up creating more work f…