The case for zero-error horizons in trustworthy LLMs
1–10 of 121 posts
Re: The case for zero-error horizons in trustworthy LLMs
#2Why didn't OpenAI finetune the model to use the python tool it has for these tasks?
Re: The case for zero-error horizons in trustworthy LLMs
#3Why didn't OpenAI finetune the model to use the python tool it has for these tasks?
They do, in the paper they mention they evaluate the LLM without tools
Re: The case for zero-error horizons in trustworthy LLMs
#4> This is surprising given the excellent capabilities of GPT-5.2.
Is this seriously surprising to anyone who knows the absolute minimum about how LLMs parse and understand text?
Re: The case for zero-error horizons in trustworthy LLMs
#5> This is surprising given the excellent capabilities of GPT-5.2. Is this seriously surprising to anyone who knows the absolute minimum about how LLMs parse and understand text?
Nope.
It's only surprising to people who still think they're going to build God out of LLMs.
Re: The case for zero-error horizons in trustworthy LLMs
#6LLMs seem to me closer to Kahneman's System 1 than to System 2. When understood in this way, it is obvious why LLMs are bad at counting r's in "strawberries". But it also makes ZEH feel like it couldn't possibly be a useful metric, because it's a System 2 evaluation applied to a System 1 system.
Re: The case for zero-error horizons in trustworthy LLMs
#7> This is surprising given the excellent capabilities of GPT-5.2
The real surprise is that someone writing a paper on LLMs doesn't understand the baseline capabilities of a hallucinatory text generator (with tool use disabled).
Re: The case for zero-error horizons in trustworthy LLMs
#8One! Two! Five!
Re: The case for zero-error horizons in trustworthy LLMs
#9Ran this through Qwen3.5-397B-A17B, and the difference between 4 characters and 5 is wild to see:
> are the following parenthesis balanced? ((())))
> No, the parentheses are not balanced.
> Here is the breakdown:
Opening parentheses (: 3
Closing parentheses ): 4
> what about these? ((((())))
> Yes, the parentheses are balanced.
> Here is the breakdown:
Opening parentheses (: 5
Closing parentheses ): 5
Re: The case for zero-error horizons in trustworthy LLMs
#10People are going to misinterpret this and overgeneralize the claim. This does not say that AI isn't reliable for things. It provides a method for quantifying the reliability for specific tasks.
You wouldn't say that a human who doesn't know how to read isn't reliable in everything, just in reading.
Counting is something that even humans need to learn how to do. Toddlers also don't understand quantity. If a 2 year old is able to count to even 10 it's through memorization and not understanding. It takes them like 2 more years of learning before they're able to comprehend things like numerical correspondence. But they do still know how to do other things that aren't counting before then.