Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ontario auditors find doctors' AI note takers routinely blow basic facts

theregister.com

Post reply on HN
Print view View original

81–90 of 141 posts

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#81

Post by wagwang »

tempest_ wrote: post #47
wagwang wrote: post #31

Earlier quoted context omitted.

You can just tell the agent to do exactly that

I've had various agents backed by various models ignore the shit out of various rules and request at varying rates but they all do it. When you point it out "Oh yes, I did do that which is contrary to the rules, request .. Anyway..."

If you are on a sota model and your context window is less than 100k tokens and you don't have any vague or contradicting rules, then I've almost never seen a rule broken

The most common failure I've seen come from tools that pollute their context with crap and the llm will forget stuff or just get confused from all the irrelevant sentences; which if the report is true, is probably what these ai notetakers are guilty of. This problem gets exacerbated if these tools turn on the 1M context window version.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#82

Post by D-Machine »

adastra22 wrote: post #34
cmrdporcupine wrote: post #25

Earlier quoted context omitted.

That's just not how they work, really. They don't know what they don't know and their process requires an output. I think they're getting better at it, but it's likely just the number of parameters getting bigger and bigger in the SOTA models more than anything.

They do know what they don't know. There's a probability distribution for outputs that they are sampling from. That just isn't being used for that purpose.

Common misconception. As far we know, LLMs are not calibrated, i.e. their output "probabilities" are not in fact necessarily correlated with the actual error rates, so you can't use e.g. the softmax values to estimate confidence. It is why it is more accurate to talk about e.g. the model "logits", "softmax values", "simplex mapping", "pseudo-probabilities", or even more agnostically, just "output scores", unless you actually have strong evidence of calibration.

To get calibrated probabilities, you actually need to use calibration techniques, and it is extremely unclear if any frontier models are doing this (or even how calibration can be done effectively in fancy chain-of-thought + MoE models, and/or how to do this in RLVR and RLHF based training regimes). I suppose if you get into things like conformal prediction, you could ensure some calibration, but this is likely too computationally expensive and/or has other undesirable side-effects.

EDIT: Oh and also there are anomaly detection approaches, which attempt to identify when we are in outlier space based on various (e.g. distance) metrics based on the embeddings, but even getting actual probabilities here is tricky. This is why it is so hard to get models to say they "don't know" with any kind of statistical certainty, because that information isn't generally actually "there" in the model, in any clean sense.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#83

Post by alterom »

autoexec wrote: post #68
alterom wrote: post #54

Earlier quoted context omitted.

>At the same time, do you really want every conversation you have with your doctor recorded Yes. This is what medical records are. They've been kept by doctors for a reason. It's not like the doctor is talking to you about which anime series are the best. You're talking about your health, your body, your disease, your treatment. It's important to keep track of that. >Plus what doctor has time to sit down and re-liste…

> Yes. This is what medical records are. They've been kept by doctors for a reason. Not every conversation. Historically, one of the nice things about doctors is that they're the ones filtering what gets included in your medical record. They decide what is medically relevant and what can remain confidential. Doctors understand that not everything discussed needs to be included in your file. Sometimes that really is j…

>Not every conversation. Historically, one of the nice things about doctors is that they're the ones filtering what gets included in your medical record.

We're in complete agreement here.

If we're not talking about an audio/video recording (a thing that nobody needs), the act of producing a record of a conversation involves choosing what goes into it.

We both agree that not every words that was said needs to go there. By far.

I guess it would be correct to say that there needs to be a record of every medical visit, but nobody needs a recording.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#84

Post by adastra22 »

D-Machine wrote: post #82
adastra22 wrote: post #34

Earlier quoted context omitted.

They do know what they don't know. There's a probability distribution for outputs that they are sampling from. That just isn't being used for that purpose.

Common misconception. As far we know, LLMs are not calibrated, i.e. their output "probabilities" are not in fact necessarily correlated with the actual error rates, so you can't use e.g. the softmax values to estimate confidence. It is why it is more accurate to talk about e.g. the model "logits", "softmax values", "simplex mapping", "pseudo-probabilities", or even more agnostically, just "output scores", unless you…

I don't know if we are talking past each other, but I don't think this conversation is about absolute probabilities? The question is about relative uncertainty, and the softmax values are just fine for that.

It is too computationally expensive, which is why nobody does this for production inference. But there are alignment tools to extract out these latent-space probabilities for researchers in the frontier labs.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#85

Post by D-Machine »

fc417fc802 wrote: post #77
colechristensen wrote: post #72

Earlier quoted context omitted.

are you actually curious or do you just want to argue against it?

I think you're obviously wrong (based on my relatively detailed but certainly somewhat out of date and not expert level knowledge of LLM internals) but if you're willing to explain your reasoning I'm willing to reconsider my own position in light of any new information or novel observations you might provide.

GP is obviously wrong, and probably doesn't know about calibration and/or that it isn't even clear how to calibrate frontier models in the manner we need, given how complex and expensive the training is, and how tricky calibration becomes in e.g. mixture-of-experts and chain of thought approaches.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#86

Post by alterom »

aakresearch wrote: post #75
alterom wrote: post #54

Earlier quoted context omitted.

>At the same time, do you really want every conversation you have with your doctor recorded Yes. This is what medical records are. They've been kept by doctors for a reason. It's not like the doctor is talking to you about which anime series are the best. You're talking about your health, your body, your disease, your treatment. It's important to keep track of that. >Plus what doctor has time to sit down and re-liste…

>>At the same time, do you really want every conversation you have with your doctor recorded >Yes. This is what medical records are. No. Medical records are limited extracts from conversations , which is your doctor and only your doctor is qualified to make, using "semantic analysis applied to your unique situation", not "linguistic probabilistic inference applied to conversation about your situation using token weig…

Hey, I'm agreement with you.

I meant that these limited extracts do need to be recorded, that's all.

Read the rest of the comment :)

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#87

Post by D-Machine »

adastra22 wrote: post #84
D-Machine wrote: post #82

Earlier quoted context omitted.

Common misconception. As far we know, LLMs are not calibrated, i.e. their output "probabilities" are not in fact necessarily correlated with the actual error rates, so you can't use e.g. the softmax values to estimate confidence. It is why it is more accurate to talk about e.g. the model "logits", "softmax values", "simplex mapping", "pseudo-probabilities", or even more agnostically, just "output scores", unless you…

I don't know if we are talking past each other, but I don't think this conversation is about absolute probabilities? The question is about relative uncertainty, and the softmax values are just fine for that. It is too computationally expensive, which is why nobody does this for production inference. But there are alignment tools to extract out these latent-space probabilities for researchers in the frontier labs.

> The question is about relative uncertainty, and the softmax values are just fine for that.

They really aren't, especially if you consider the chain of thought / recursive application case, and also that you can't even assume e.g. a difference of 0.1 in softmax values means the same relative difference from input to input, or that e.g. an 0.9 is always "extremely confident", and etc. You really have no idea unless you are testing the calibration explicitly on calibration data.

> But there are alignment tools to extract out these latent-space probabilities for researchers in the frontier labs

You can get embeddings: if you can get calibrated probabilities, you'll need to provide a citation, because this would be a huge deal for all sorts of applications.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#88

Post by adastra22 »

chongli wrote: post #67
adastra22 wrote: post #34

Earlier quoted context omitted.

They do know what they don't know. There's a probability distribution for outputs that they are sampling from. That just isn't being used for that purpose.

Having a probability distribution to sample from is not the same thing is knowing, because they don’t know anything about the provenance of the data that was used to build the distribution. They trust their training set implicitly by construction. They have no means to detect systematic errors in their training set.

You are talking about something different. If I ask you a yes/no question, and then ask you how certain you are, the answer you give is not an objective measurement of how likely you are to be right. You don't have access to that either. If you say "I'm very confident" or "Maybe 50/50" -- that is an assessment of your own internal weighted evidence, which is the equivalent of an LLM's softmax distribution.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#89

Post by adastra22 »

D-Machine wrote: post #87
adastra22 wrote: post #84

Earlier quoted context omitted.

I don't know if we are talking past each other, but I don't think this conversation is about absolute probabilities? The question is about relative uncertainty, and the softmax values are just fine for that. It is too computationally expensive, which is why nobody does this for production inference. But there are alignment tools to extract out these latent-space probabilities for researchers in the frontier labs.

> The question is about relative uncertainty, and the softmax values are just fine for that. They really aren't, especially if you consider the chain of thought / recursive application case, and also that you can't even assume e.g. a difference of 0.1 in softmax values means the same relative difference from input to input, or that e.g. an 0.9 is always "extremely confident", and etc. You really have no idea unless y…

Relative probabilities. That means comparing 2+ alternatives, and we're only talking about the model's worldview, not objective reality. The math for that is relatively straightforward. "Yes" could be 0.9, and ok that means nothing. But If we artificially constraint outputs to "Yes" and "No", and calculate the softmax for Yes to be 0.7 and No to be 0.3, that does lead to a straightforward probability calculation. [Not the naïve calculation you would expect, because of how softmax is computed. But you can derive an equation to convert it into normalized probabilities.]

And now I'm certain we're taking past each other. I'm not talking about calibrated probabilities at all. Just the notion of "how confident do I feel about this?" which is what I interpreted the question above to be about. You can get that out of an LLM, with some work.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#90

Post by jcgrillo »

cootsnuck wrote: post #69
rainsford wrote: post #11

I have generally moved from bearish to bullish on the future of current AI technology, but the continued inaccuracy with basic facts all while the models significantly improve continues to give me significant pause. As an example, creating recipes with Claude Opus based on flavor profiles and preferences feels magical, right up until the point at which it can't accurately convert between tablespoons and teaspoons. It…

Yup, spot on. There's a capability-reliability gap that the industry does not like to talk about too much. It often feels like the AI industry is continually glossing over the fact that capability and reliability are fundamentally different qualities. We tend to use "accurate" and "reliable" interchangeably, but they describe different things. A model can ace a benchmark (capability/accuracy) and still be a liability…

This capability-reliability gap (excellent term btw, more people need to think in these terms or we'll be in real trouble) is also infecting LLM assisted outputs. I just tried VSCode again tonight after a ~3yr hiatus and goddamn has it deteriorated. Lots of new features, lots of interesting looking plugins, but 3 out of the 5 plugins I tried for code CAD (the reason I downloaded VSCode again at all) were completely unusable--like couldn't even be made to work at all--and the other two didn't do anything like what they claimed. Also VSCode itself got into some kind of spastic loop trying to log me into github, and seemed incapable of recognizing the virtual environment in a python project's workspace... It also feels like the UI got even slower. This situation is bad.
Post reply on HN