Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ontario auditors find doctors' AI note takers routinely blow basic facts

theregister.com

Post reply on HN
Print view View original

31–40 of 141 posts

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#31

Post by wagwang »

themafia wrote: post #16
rainsford wrote: post #11

I have generally moved from bearish to bullish on the future of current AI technology, but the continued inaccuracy with basic facts all while the models significantly improve continues to give me significant pause. As an example, creating recipes with Claude Opus based on flavor profiles and preferences feels magical, right up until the point at which it can't accurately convert between tablespoons and teaspoons. It…

> we're not actually on the right track to achieve real intelligence. Real intelligence means you have to say "I don't know" when you don't know, or ask for help, or even just saying you refuse to help with the subtext being you don't want to appear stupid. The models could ostensibly do this when it has low confidence in it's own results but they don't. What I don't know if it's because it would be very computationa…

You can just tell the agent to do exactly that

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#32

Post by skydhash »

colechristensen wrote: post #30
mylifeandtimes wrote: post #26

Earlier quoted context omitted.

could you please tell me how it generates that certainty score?

The whole thing is a statistical model, that's just what it is. No, I cannot in a reasonable way dissect how an LLM works to a satisfactory level to a skeptic.

It's a statistical model for words and sentences, not knowledge. What does the LLM knows about having a pebble in your shoes, or drinking a nice cup of coffee?

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#33

Post by dmix »

> They specifically address the AI Scribe program, the Ontario Ministry of Health initiated for physicians, nurse practitioners, and other healthcare professionals across the broader health sector.

makes me wonder what quality software the ministry would push (probably mostly qualifications like SOC).

This is apparently this list of approved vendors

https://www.supplyontario.ca/vor/software/tender-20123-artif...

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#34

Post by adastra22 »

cmrdporcupine wrote: post #25
themafia wrote: post #16

Earlier quoted context omitted.

> we're not actually on the right track to achieve real intelligence. Real intelligence means you have to say "I don't know" when you don't know, or ask for help, or even just saying you refuse to help with the subtext being you don't want to appear stupid. The models could ostensibly do this when it has low confidence in it's own results but they don't. What I don't know if it's because it would be very computationa…

That's just not how they work, really. They don't know what they don't know and their process requires an output. I think they're getting better at it, but it's likely just the number of parameters getting bigger and bigger in the SOTA models more than anything.

They do know what they don't know. There's a probability distribution for outputs that they are sampling from. That just isn't being used for that purpose.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#35

Post by adastra22 »

mylifeandtimes wrote: post #26
colechristensen wrote: post #22

Earlier quoted context omitted.

You can TELL the models to do this and they'll follow your prompt. "Give me your answer and rate each part of it for certainty by percentage" or similar.

could you please tell me how it generates that certainty score?

Vibes.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#36

Post by vor_ »

ceejayoz wrote: post #3

> 60% of evaluated AI Scribe systems mixed up prescribed drugs in patient notes, auditors say Not mentioned, as far as I can see: the comparative human mistake rate. Having seen a lot of medical records, 60% sounds about normal lol.

60% is a normal human mistake rate? You can't be serious.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#37

Post by NateEag »

jmward01 wrote: post #12
ceejayoz wrote: post #3

> 60% of evaluated AI Scribe systems mixed up prescribed drugs in patient notes, auditors say Not mentioned, as far as I can see: the comparative human mistake rate. Having seen a lot of medical records, 60% sounds about normal lol.

This is not a popular view 'AI sucks at X but so do humans' but I think it is valid and we should take wins where we can, especially in healthcare. It is pretty clear that initial accuracy issues will become less and less of a problem as these technologies mature. This focus on accuracy now as a 'see it's bad' talking point though misses the real danger. Medical note takers have an exceptionally high chance of being…

> It is pretty clear that initial accuracy issues will become less and less of a problem as these technologies mature.

What do you base this on?

As someone who can both see the amazing things genAI can do, and who sees how utterly flawed most genAI output is, it's not obvious to me.

I'm working with Claude every day, Opus 4.7, and reviewing a steady stream of PRs from coworkers who are all-in, not just using due to corporate mandates like me, and I find an unending stream of stupidity and incomprehension from these bots that just astonishes me.

Claude recently output this to me:

"I've made those changes in three files:

- File 1

- File 2"

That is a vintage hallucination that could've come right out of GPT 2.0.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#38

Post by Ferret7446 »

zOneLetter wrote: post #2

Anecdotally, we use an LLM note-taker at work for meetings. I had to intervene recently because our CIO was VERY angry at our vendor for something they promised to do and never did. He wasn't at the meeting where the "promise" was made. I was. They never promised anything, and the discussion was significantly more nuanced than what the LLM wrote in the detailed summary. In other cases, I have seen it miss the mark wh…

Transcription works pretty well in my experience, and the transcripts should be treated as the ground truth in such cases.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#39

Post by fc417fc802 »

zOneLetter wrote: post #2

Anecdotally, we use an LLM note-taker at work for meetings. I had to intervene recently because our CIO was VERY angry at our vendor for something they promised to do and never did. He wasn't at the meeting where the "promise" was made. I was. They never promised anything, and the discussion was significantly more nuanced than what the LLM wrote in the detailed summary. In other cases, I have seen it miss the mark wh…

> I would think for compliance reasons hospitals would not want to alter the records and only go by transcripts, but what do I know...

I'm puzzled by this as well. Why not just generate a transcript and be done with it? If it's a particularly long transcript that's being referenced repeatedly for whatever reason let the humans manually mark it up with a side by side summary when and where they feel the need. At least my experience is that usually these sort of interactions don't have a lot of extraneous data that can be casually filtered out to begin with. The details tend to matter quite a lot!

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#40

Post by alterom »

wagwang wrote: post #31
themafia wrote: post #16

Earlier quoted context omitted.

> we're not actually on the right track to achieve real intelligence. Real intelligence means you have to say "I don't know" when you don't know, or ask for help, or even just saying you refuse to help with the subtext being you don't want to appear stupid. The models could ostensibly do this when it has low confidence in it's own results but they don't. What I don't know if it's because it would be very computationa…

You can just tell the agent to do exactly that

>You can just tell the agent to do exactly that

You can.

It just won't do it.

Post reply on HN