Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI agent benchmarks are broken

ddkang.substack.com

Post reply on HN
Print view View original

81–88 of 88 posts

Re: AI agent benchmarks are broken

#81

Post by Jensson »

brookst wrote: post #69
rsynnott wrote: post #28

Earlier quoted context omitted.

... I mean, when evaluating "45 + 8 minutes" where the expected answer was "63 minutes", as in the article, a competent human reviewer does not go "hmm, yes, that seems plausible, it probably succeeded, give it the points". I know LLM evangelists love this "humans make mistakes too" line, but, really, only an _exceptionally_ incompetent human evaluator would fall for that one.

have you ever hired human evaluators at scale? They make all sorts of mistakes. Relatively low probability, so it’s a noise factor in, but I have yet to meet the human who is 100% accurate at simple tasks done thousands of times.

Which is why you hire them at scale as you say, then they are very reliable. LLM at scale are not.

The problem with these AI models is there is no such point where you can just scale them up and they can solve problems as accurately as a group of humans. They add too much noise and eventually go haywire when left to their own devices.

Re: AI agent benchmarks are broken

#82

Post by ggregoryarms »

mycall wrote: post #10

SnitchBench [0] is unique benchmark which shows how aggressively models will snitch on you via email and CLI tools when they are presented with evidence of corporate wrongdoing - measuring their likelihood to "snitch" to authorities. I don't believe they were trained to do this, so it seems to be an emergent ability. [0] https://snitchbench.t3.gg/

Seems like more of a subtextual/accidental ability than an emergent ability.

Re: AI agent benchmarks are broken

#83

Post by e1g »

brookst wrote: post #65
e1g wrote: post #30

Earlier quoted context omitted.

Agree, current "thinking" models are effectively "re-run this question N times, and determine the best answer", and this LLM-evaluating-LLM loop demonstrably leads to higher quality answers against objective metrics (in math, etc).

That’s… not how thinking models work. They tend to be iterative and serial, not parallel and then pick-one.

Parallel test time compute is exactly what SOTA models do, including Claude 4 Opus extended, o3 Pro, Grok 4 Heavy, and Gemini 2.5 Pro.

Re: AI agent benchmarks are broken

#84

Post by brookst »

Jensson wrote: post #81
brookst wrote: post #69

Earlier quoted context omitted.

have you ever hired human evaluators at scale? They make all sorts of mistakes. Relatively low probability, so it’s a noise factor in, but I have yet to meet the human who is 100% accurate at simple tasks done thousands of times.

Which is why you hire them at scale as you say, then they are very reliable. LLM at scale are not. The problem with these AI models is there is no such point where you can just scale them up and they can solve problems as accurately as a group of humans. They add too much noise and eventually go haywire when left to their own devices.

I haven’t found that to be the case. Both LLMs and humans produce outputs that cannot be blindly trusted to be accurate.

Re: AI agent benchmarks are broken

#85

Post by ttoinou »

th0ma5 wrote: post #78
ttoinou wrote: post #23

What makes LLMs amazing (fuzzy input, fuzzy output) is exactly why they are hard to benchmark. If they could be benchmarked easily, they wouldn't be powerful by definition. I have no idea what's going on in the minds of people benchmarking LLMs for fuzzy tasks, and in the minds of people relying on benchmarks to make decisions about LLMs, I never looked at them. People doing benchmarks have to prove what they do is u…

Since when do people like the fuzziness of outputs? I think you make an interesting point but it also seems to imply that benchmarking will never truly be possible, which I think is true unless we can also make them observable which also as you say gives up the mystique.

There is also the possibility that LLM might help us understand language better and make this domain more rigorous. Only researchers can see if thats gonna be happening

  Since when do people like the fuzziness of outputs?
A lot of tasks are fuzzy by nature, there are multiple valid results, multiple interpretation of the situation / context etc. We’re gonna discover new areas where computers will be useful finally

Re: AI agent benchmarks are broken

#86

Post by tempfile »

rf15 wrote: post #64
tempfile wrote: post #48

Earlier quoted context omitted.

> Discriminating good answers is easier than generating them. This is actually very wrong. Consider for instance the fact that people who grade your tests in school are typically more talented, capable, trained than the people taking the test. This is true even when an answer key exists. > Also, human labels are good but have problems of their own, Granted, but... > it isn’t like by using a “different intelligence ar…

Trading control for convenience has always been the tradeoff in the recent AI hype cycle and the reason why so many people like to use ChatGPT.

Not "control", "a control". As in a control group, for a study.

Re: AI agent benchmarks are broken

#87

Post by jacobr1 »

brookst wrote: post #68
jacobr1 wrote: post #47

Earlier quoted context omitted.

The equivalent would be having the _same human_ review their own work. We require others with different experience and fresh eyes for secondary review and for the most important task multiple people. To some extent the same llm with a new context history and different prompt is sorta like that ... but still is much weaker than using a different system entirely.

How do you feel about o3 reviewing 4o-mini?

Might be ok. I'm in the camp right now that you need run evals, test, and iterate to maximize your performance. But everything is going depend on the specifics of your use case - and that should be assessed empirically.

Re: AI agent benchmarks are broken

#88

Post by tempfile »

alextheparrot wrote: post #73
tempfile wrote: post #48

Earlier quoted context omitted.

> Discriminating good answers is easier than generating them. This is actually very wrong. Consider for instance the fact that people who grade your tests in school are typically more talented, capable, trained than the people taking the test. This is true even when an answer key exists. > Also, human labels are good but have problems of their own, Granted, but... > it isn’t like by using a “different intelligence ar…

It isn’t actually very wrong. Your example is tangential as graders in school have multiple roles — teaching the content and grading. That’s an implementation detail, not a counter to the premise. I don’t think we should assume answering a test would be easy for a Scantron machine just because it is very good at grading them, either.

No. Graders having multiple roles is actually the implementation detail, since they're people, and they can't spend all day grading work. Scanning machines don't really grade work either, but I am happy to rely on them for checking an answer matches a scheme verbatim. I'm not sure why you mention scanners answering tests either, since my original comment doesn't imply that.

There is no evidence that an LLM can reliably evaluate the semantic content of a sentence, even in cases where we all agree that the semantic content exists. The thread we are participating in demonstrates a particularly egregious failure, but there is no good reason to think that more subtle failures might not exist if we happen to patch this one. Even if they were reliable, you can't evaluate a system with itself - that is basic science.

Post reply on HN