[flagged]
Ontario auditors find doctors' AI note takers routinely blow basic facts
21–30 of 141 posts
Re: Ontario auditors find doctors' AI note takers routinely blow basic facts
#22I have generally moved from bearish to bullish on the future of current AI technology, but the continued inaccuracy with basic facts all while the models significantly improve continues to give me significant pause. As an example, creating recipes with Claude Opus based on flavor profiles and preferences feels magical, right up until the point at which it can't accurately convert between tablespoons and teaspoons. It…
> we're not actually on the right track to achieve real intelligence. Real intelligence means you have to say "I don't know" when you don't know, or ask for help, or even just saying you refuse to help with the subtext being you don't want to appear stupid. The models could ostensibly do this when it has low confidence in it's own results but they don't. What I don't know if it's because it would be very computationa…
You can TELL the models to do this and they'll follow your prompt.
"Give me your answer and rate each part of it for certainty by percentage" or similar.
Re: Ontario auditors find doctors' AI note takers routinely blow basic facts
#23Earlier quoted context omitted.
I hate to help provide possible soultions to an entire process I don't approve of, but maybe the fuzzy tools need old style deterministic tools the same way and for the same reasons we do. So instead of an LLM trying to answer a math or reason question by finding a statistical match with other similar groups of words it found on 4chan and the all in podcast and a terrible recipe for soup written by a terrible cook, i…
No, they just need to be trained to have adversarial self review "thinking" processes. You ask an LLM "What's wrong with your answer?" and you get pretty good results.
Or you get the original output result was perfect and the adversarial "rethinking" switches to an incorrect result.
Re: Ontario auditors find doctors' AI note takers routinely blow basic facts
#24The AI note taker we use at work records the meeting as well, and each note it takes about the meeting has a timestamp link that takes you directly there in the recording so you can check it yourself. While I'm sure a solution like this is more complicated in a HIPPAA environment, something like this is critical for things as important as healthcare.
When designing AI-based user experiences I refer to this as provenance. It’s a vital aspect of trust, reliability, compliance and more. If a software system includes LLM output like this but doesn’t surface the provenance of its output for human evaluation and verification then it’s at best poor user experience, and at worst a dangerous one.
At the same time, do you really want every conversation you have with your doctor recorded, handed over to third party companies, and stored forever with your medical file? Plus what doctor has time to sit down and re-listen to your visit to check to make sure the AI didn't screw up at some point in the future anyway? If your doctor isn't going to be verifying the accuracy from those recordings who would? Overseas contractors? At what point does it become a larger waste of time and money to babysit an incompetent AI than just not using one in the first place?
There are some good uses for AI, but I'm not convinced that this (or many other cases where accuracy matters) is one of them.
Re: Ontario auditors find doctors' AI note takers routinely blow basic facts
#25I have generally moved from bearish to bullish on the future of current AI technology, but the continued inaccuracy with basic facts all while the models significantly improve continues to give me significant pause. As an example, creating recipes with Claude Opus based on flavor profiles and preferences feels magical, right up until the point at which it can't accurately convert between tablespoons and teaspoons. It…
> we're not actually on the right track to achieve real intelligence. Real intelligence means you have to say "I don't know" when you don't know, or ask for help, or even just saying you refuse to help with the subtext being you don't want to appear stupid. The models could ostensibly do this when it has low confidence in it's own results but they don't. What I don't know if it's because it would be very computationa…
That's just not how they work, really. They don't know what they don't know and their process requires an output.
I think they're getting better at it, but it's likely just the number of parameters getting bigger and bigger in the SOTA models more than anything.
Re: Ontario auditors find doctors' AI note takers routinely blow basic facts
#26Earlier quoted context omitted.
> we're not actually on the right track to achieve real intelligence. Real intelligence means you have to say "I don't know" when you don't know, or ask for help, or even just saying you refuse to help with the subtext being you don't want to appear stupid. The models could ostensibly do this when it has low confidence in it's own results but they don't. What I don't know if it's because it would be very computationa…
You can TELL the models to do this and they'll follow your prompt. "Give me your answer and rate each part of it for certainty by percentage" or similar.
could you please tell me how it generates that certainty score?
Re: Ontario auditors find doctors' AI note takers routinely blow basic facts
#27> 60% of evaluated AI Scribe systems mixed up prescribed drugs in patient notes, auditors say Not mentioned, as far as I can see: the comparative human mistake rate. Having seen a lot of medical records, 60% sounds about normal lol.
Even if you had the same 60% error rate with humans the types of errors would be vastly different. Humans might make typos, or forget to include something, or even occasionally misremember some minor detail, but that's very different from BS AI just hallucinates out of nowhere. AI makes the kinds of mistakes no human ever would which means they can be extremely confusing and easy to catch or they can be something no human would even think to question or be looking out for because it makes no sense why AI would randomly (and confidently) say something so wrong.
Re: Ontario auditors find doctors' AI note takers routinely blow basic facts
#28And once again, we have an example of how AI is a liability issue waiting to happen.
Re: Ontario auditors find doctors' AI note takers routinely blow basic facts
#29> 60% of evaluated AI Scribe systems mixed up prescribed drugs in patient notes, auditors say Not mentioned, as far as I can see: the comparative human mistake rate. Having seen a lot of medical records, 60% sounds about normal lol.
This is not a popular view 'AI sucks at X but so do humans' but I think it is valid and we should take wins where we can, especially in healthcare. It is pretty clear that initial accuracy issues will become less and less of a problem as these technologies mature. This focus on accuracy now as a 'see it's bad' talking point though misses the real danger. Medical note takers have an exceptionally high chance of being…
> It is pretty clear that initial accuracy issues will become less and less of a problem as these technologies mature.
Does it?
Re: Ontario auditors find doctors' AI note takers routinely blow basic facts
#30Earlier quoted context omitted.
You can TELL the models to do this and they'll follow your prompt. "Give me your answer and rate each part of it for certainty by percentage" or similar.
could you please tell me how it generates that certainty score?
The whole thing is a statistical model, that's just what it is. No, I cannot in a reasonable way dissect how an LLM works to a satisfactory level to a skeptic.