Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ontario auditors find doctors' AI note takers routinely blow basic facts

theregister.com

Post reply on HN
Print view View original

71–80 of 141 posts

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#71

Post by analog31 »

Brian_K_White wrote: post #18
rainsford wrote: post #11

I have generally moved from bearish to bullish on the future of current AI technology, but the continued inaccuracy with basic facts all while the models significantly improve continues to give me significant pause. As an example, creating recipes with Claude Opus based on flavor profiles and preferences feels magical, right up until the point at which it can't accurately convert between tablespoons and teaspoons. It…

I hate to help provide possible soultions to an entire process I don't approve of, but maybe the fuzzy tools need old style deterministic tools the same way and for the same reasons we do. So instead of an LLM trying to answer a math or reason question by finding a statistical match with other similar groups of words it found on 4chan and the all in podcast and a terrible recipe for soup written by a terrible cook, i…

Doesn't agentic AI do this? I've got AI running in VS Code. If I ask it for something, it can fill a code cell with a little bit of Python, and then run it with my approval. It's using the Python interpreter on my computer as a calculator.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#72

Post by colechristensen »

fc417fc802 wrote: post #43
colechristensen wrote: post #30

Earlier quoted context omitted.

The whole thing is a statistical model, that's just what it is. No, I cannot in a reasonable way dissect how an LLM works to a satisfactory level to a skeptic.

He's not a skeptic, he's asking you to explicitly state your reasoning with the expectation that either the readers will learn something or (more likely) you will realize that your thought and speech pattern there was the equivalent of an LLM hallucinating. Yes you can prompt it as you suggested and yes you will generally receive a convincing answer but it is not doing what you seem to think it is doing ie the genera…

are you actually curious or do you just want to argue against it?

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#73

Post by clipsy »

colechristensen wrote: post #72
fc417fc802 wrote: post #43

Earlier quoted context omitted.

He's not a skeptic, he's asking you to explicitly state your reasoning with the expectation that either the readers will learn something or (more likely) you will realize that your thought and speech pattern there was the equivalent of an LLM hallucinating. Yes you can prompt it as you suggested and yes you will generally receive a convincing answer but it is not doing what you seem to think it is doing ie the genera…

are you actually curious or do you just want to argue against it?

"I can only explain my beliefs to people who promise they'll agree" is certainly a unique take.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#74

Post by cootsnuck »

LAC-Tech wrote: post #61

Can someone who is a more AI heavy user explain what is going on? I would expect an "AI Note Taker" to faithfully transcribe the entire conversation. With the same quality I see in a lot of automated video subtitles.. ie they use the wrong word a lot but it's easy to tell what they mean by context. Are these tools instead immediately summarising the whole thing, and that summary is the artifact? Because that is a bey…

I work specifically in voice AI and am very familiar with how these tools and systems work.

> I would expect an "AI Note Taker" to faithfully transcribe the entire conversation. With the same quality I see in a lot of automated video subtitles.. ie they use the wrong word a lot but it's easy to tell what they mean by context.

That's a reasonable expectation, but would not be a safe one. All transcription tools are not made the same. First it depends on what kind of STT/ASR (speech-to-text / automatic speech recognition) model they are using. A lot of tools like to use some flavor of OpenAI's Whisper model. It works well generally but I would never use it in a critical use case like healthcare. Because it can hallucinate. That's specific to its architecture and how it was trained.

There's a fairly large variety of architectures that can be used for STT/ASR. Some of them are designed for "offline" / "batch" / pre-recorded audio. Some are designed for fast real-time streaming transcription.

There are more factors too like training data. And not just demographics of the speakers in the training data but audio environments too. Was the model trained on echo-y doctor offices with two people being recorded from a crappy smartphone mic or desktop mic? (It could've been! But it's an important distinction.)

And there's more factors than that, but you get the picture (e.g. are they trying to "clean up" the transcript afterwards by feeding it to an LLM, are they attempting to pre-process audio before transcription also in an attempt to boost accuracy)

There's a lot of ways to do it, meaning, there's a lot of ways to screw it up.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#75

Post by aakresearch »

alterom wrote: post #54
autoexec wrote: post #24

Earlier quoted context omitted.

At the same time, do you really want every conversation you have with your doctor recorded, handed over to third party companies, and stored forever with your medical file? Plus what doctor has time to sit down and re-listen to your visit to check to make sure the AI didn't screw up at some point in the future anyway? If your doctor isn't going to be verifying the accuracy from those recordings who would? Overseas co…

>At the same time, do you really want every conversation you have with your doctor recorded Yes. This is what medical records are. They've been kept by doctors for a reason. It's not like the doctor is talking to you about which anime series are the best. You're talking about your health, your body, your disease, your treatment. It's important to keep track of that. >Plus what doctor has time to sit down and re-liste…

>>At the same time, do you really want every conversation you have with your doctor recorded

>Yes. This is what medical records are.

No. Medical records are limited extracts from conversations, which is your doctor and only your doctor is qualified to make, using "semantic analysis applied to your unique situation", not "linguistic probabilistic inference applied to conversation about your situation using token weights averaged over billion unrelated samples"

> It's not like the doctor is talking to you about which anime series are the best. You're talking about your health, your body, your disease, your treatment.

No jokes, no banter, no chit-chat, no complements to doctor's new Tesla?

> It's important to keep track of that.

Same fallacy Meta fell into when started tracking employees' keystrokes and mouse gestures. 90% of my mouse movements are just fidgeting, with no relation to the task at hand - and it is not a crime! But if I knew my mouse fidgeting is being watched, I'll make sure that percentage goes up to 99% - for the LLM which is gonna be trained off it to self-immolate over its NSFW nature.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#76

Post by MintPaw »

toraway wrote: post #46
zOneLetter wrote: post #2

Anecdotally, we use an LLM note-taker at work for meetings. I had to intervene recently because our CIO was VERY angry at our vendor for something they promised to do and never did. He wasn't at the meeting where the "promise" was made. I was. They never promised anything, and the discussion was significantly more nuanced than what the LLM wrote in the detailed summary. In other cases, I have seen it miss the mark wh…

I recently left my mom a voicemail saying happy Mother’s Day with normal human boilerplate of sorry I missed you, feel free to give me a call back tonight or we can talk tomorrow, either is fine by me whatever works best for you, hope we can talk soon, love you, bye. She called me back later that night and we chatted for bit and then she paused and sort of uncertainly was like “So… was there something you were needin…

What is a voicemail in this context? What app is reading it?

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#77

Post by fc417fc802 »

colechristensen wrote: post #72
fc417fc802 wrote: post #43

Earlier quoted context omitted.

He's not a skeptic, he's asking you to explicitly state your reasoning with the expectation that either the readers will learn something or (more likely) you will realize that your thought and speech pattern there was the equivalent of an LLM hallucinating. Yes you can prompt it as you suggested and yes you will generally receive a convincing answer but it is not doing what you seem to think it is doing ie the genera…

are you actually curious or do you just want to argue against it?

I think you're obviously wrong (based on my relatively detailed but certainly somewhat out of date and not expert level knowledge of LLM internals) but if you're willing to explain your reasoning I'm willing to reconsider my own position in light of any new information or novel observations you might provide.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#78

Post by DANmode »

Arodex wrote: post #4
ceejayoz wrote: post #3

> 60% of evaluated AI Scribe systems mixed up prescribed drugs in patient notes, auditors say Not mentioned, as far as I can see: the comparative human mistake rate. Having seen a lot of medical records, 60% sounds about normal lol.

But who is responsible is different. (And if you already see 60% error rates in standard, pre-AI note taking, how does that not translate into many deaths and injury? At least one country's health system in the world should have caught that)

The human who hits Submit or Approve is responsible.

The management human who offered the bad tool to the other human is responsible.

The robot cannot be responsible in place of us.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#80

Post by wagwang »

alterom wrote: post #40
wagwang wrote: post #31

Earlier quoted context omitted.

You can just tell the agent to do exactly that

>You can just tell the agent to do exactly that You can. It just won't do it .

Seems to work for me

https://chatgpt.com/share/6a06a4c5-d454-83e8-a5b2-c9468f6588...

Post reply on HN