The "confident idiot" problem: Why AI needs hard rules, not vibe checks
steerlabs.substack.com
The "confident idiot" problem: Why AI needs hard rules, not vibe checks
1–10 of 399 posts
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#2We are trying to fix probability with more probability. That is a losing game.
Thanks for pointing out the elephant in the room with LLMs.
The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#3OP here. I wrote this because I got tired of agents confidently guessing answers when they should have asked for clarification (e.g. guessing "Springfield, IL" instead of asking "Which state?" when asked "weather in Springfield").
I built an open-source library to enforce these logic/safety rules outside the model loop: https://github.com/imtt-dev/steer
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#4We are trying to fix probability with more probability. That is a losing game. Thanks for pointing out the elephant in the room with LLMs. The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility.
Exactly. We treat them like databases, but they are hallucination machines.
My thesis isn't that we can stop the hallucinating (non-determinism), but that we can bound it.
If we wrap the generation in hard assertions (e.g., assert response.price > 0), we turn 'probability' into 'manageable software engineering.' The generation remains probabilistic, but the acceptance criteria becomes binary and deterministic.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#5We are trying to fix probability with more probability. That is a losing game. Thanks for pointing out the elephant in the room with LLMs. The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility.
Exactly. We treat them like databases, but they are hallucination machines. My thesis isn't that we can stop the hallucinating (non-determinism), but that we can bound it. If we wrap the generation in hard assertions (e.g., assert response.price > 0), we turn 'probability' into 'manageable software engineering.' The generation remains probabilistic, but the acceptance criteria becomes binary and deterministic.
but the acceptance criteria becomes binary and deterministic.
Unfortunately, the use-case for AI is often where the acceptance criteria is not easily defined --- a matter of judgment. For example, "Does this patient have cancer?".
In cases where the criteria can be easily and clearly stipulated, AI often isn't really required.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#6Earlier quoted context omitted.
Exactly. We treat them like databases, but they are hallucination machines. My thesis isn't that we can stop the hallucinating (non-determinism), but that we can bound it. If we wrap the generation in hard assertions (e.g., assert response.price > 0), we turn 'probability' into 'manageable software engineering.' The generation remains probabilistic, but the acceptance criteria becomes binary and deterministic.
but the acceptance criteria becomes binary and deterministic. Unfortunately, the use-case for AI is often where the acceptance criteria is not easily defined --- a matter of judgment. For example, "Does this patient have cancer?". In cases where the criteria can be easily and clearly stipulated, AI often isn't really required.
You're 100% right. For a "judgment" task like "Does this patient have cancer?", the final acceptance criteria must be a human expert. A purely deterministic verifier is impossible.
My thesis is that even in those "fuzzy" workflows, the agent's process is full of small, deterministic sub-tasks that can and should be verified.
For example, before the AI even attempts to analyze the X-ray for cancer, it must: 1/ Verify it has the correct patient file (PatientIDVerifier). 2/ Verify the image is a chest X-ray and not a brain MRI (ModalityVerifier). 3/ Verify the date of the scan is within the relevant timeframe (DateVerifier).
These are "boring," deterministic checks. But a failure on any one of them makes the final "judgment" output completely useless.
steer isn't designed to automate the final, high-stakes judgment. It's designed to automate the pre-flight checklist, ensuring the agent has the correct, factually grounded information before it even begins the complex reasoning task. It's about reducing the "unforced errors" so the human expert can focus only on the truly hard part.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#7[dead]
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#8We are trying to fix probability with more probability. That is a losing game. Thanks for pointing out the elephant in the room with LLMs. The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility.
Exactly. We treat them like databases, but they are hallucination machines. My thesis isn't that we can stop the hallucinating (non-determinism), but that we can bound it. If we wrap the generation in hard assertions (e.g., assert response.price > 0), we turn 'probability' into 'manageable software engineering.' The generation remains probabilistic, but the acceptance criteria becomes binary and deterministic.
> We treat them like databases, but they are hallucination machines.
Which is kind of crazy because we don't even treat people as databases. Or at least we shouldn't.
Maybe it's one of those things that will disappear form culture one funeral at a time.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#9What I do, is actually running the task. If it is script, getting logs. If it is is website, getting screenshots. Otherwise it is coding in the blind.
Alike writing a script and having the attitude "yeah, I am good at it, I don't need to actually run it to know if works" - well, likely, it won't work. Maybe because of a trivial mistake.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#10We are trying to fix probability with more probability. That is a losing game. Thanks for pointing out the elephant in the room with LLMs. The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility.
lol humans are non-deterministic too