Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI agent benchmarks are broken

ddkang.substack.com

Post reply on HN
Print view View original

61–70 of 88 posts

Re: AI agent benchmarks are broken

#61

Post by majormajor »

jgraettinger1 wrote: post #56
majormajor wrote: post #44

Earlier quoted context omitted.

> Discriminating good answers is easier than generating them. I don't think this is true for many fields - especially outside of math/programming. Let's say the task is "find the ten most promising energy startups in Europe." (This is essentially the sort of work I see people frequently talk about using research modes of models for here or on LinkedIn.) In ye olden days pre-LLM you'd be able to easily filter out a bu…

> You can't do that for LLM output. That's true if you're just evaluating the final answer. However, wouldn't you evaluate the context -- including internal tokens -- built by the LLM under test ? In essence, the evaluator's job isn't to do separate fact-finding, but to evaluate whether the under-test LLM made good decisions given the facts at hand.

I would if I was the developer, but if I'm the user being sold the product, or a third-party benchmarker, I don't think I'd have full access to that if most of that is happening in the vendor's internal services.

Re: AI agent benchmarks are broken

#62

Post by szvsw »

jerf wrote: post #8

When I was being a bad HN reader and just reacting to the title, my initial impulse was to be placating, and observe that they are probably just immature. After all, for all that has happened, this is still only a couple year's worth of development, and it does tend to take a long time to develop good benchmarks. However the article does seem to be pointing out some fundamental issues. I'm particularly annoyed by usi…

> I'm particularly annoyed by using LLMs to evaluate the output of LLMs.

Even though I largely agree with parts of what you wrote, if you squint your eyes enough you can kind of see an argument along the lines of “difficult to solve but easy to verify.”

Re: AI agent benchmarks are broken

#63

Post by brookst »

majormajor wrote: post #44
alextheparrot wrote: post #16

Earlier quoted context omitted.

LLMs evaluating LLM outputs really isn’t that dire… Discriminating good answers is easier than generating them. Good evaluations write test sets for the discriminators to show when this is or isn’t true. Evaluating the outputs as the user might see them are more representative than having your generator do multiple tasks (e.g. solve a math query and format the output as a multiple choice answer). Also, human labels a…

> Discriminating good answers is easier than generating them. I don't think this is true for many fields - especially outside of math/programming. Let's say the task is "find the ten most promising energy startups in Europe." (This is essentially the sort of work I see people frequently talk about using research modes of models for here or on LinkedIn.) In ye olden days pre-LLM you'd be able to easily filter out a bu…

> IMO you can never use an AI agent benchmark that is published on the internet more than once.

This is a long-solved problem far predating AI.

You do it by releasing 90% of the benchmark publicly and holding back 10% for yourself or closely trusted partners.

Then benchmark performance can be independently evaluated to determine if performance on the 10% holdback matches the 90% public.

Re: AI agent benchmarks are broken

#64

Post by rf15 »

tempfile wrote: post #48
alextheparrot wrote: post #16

Earlier quoted context omitted.

LLMs evaluating LLM outputs really isn’t that dire… Discriminating good answers is easier than generating them. Good evaluations write test sets for the discriminators to show when this is or isn’t true. Evaluating the outputs as the user might see them are more representative than having your generator do multiple tasks (e.g. solve a math query and format the output as a multiple choice answer). Also, human labels a…

> Discriminating good answers is easier than generating them. This is actually very wrong. Consider for instance the fact that people who grade your tests in school are typically more talented, capable, trained than the people taking the test. This is true even when an answer key exists. > Also, human labels are good but have problems of their own, Granted, but... > it isn’t like by using a “different intelligence ar…

Trading control for convenience has always been the tradeoff in the recent AI hype cycle and the reason why so many people like to use ChatGPT.

Re: AI agent benchmarks are broken

#65

Post by brookst »

e1g wrote: post #30
alextheparrot wrote: post #16

Earlier quoted context omitted.

LLMs evaluating LLM outputs really isn’t that dire… Discriminating good answers is easier than generating them. Good evaluations write test sets for the discriminators to show when this is or isn’t true. Evaluating the outputs as the user might see them are more representative than having your generator do multiple tasks (e.g. solve a math query and format the output as a multiple choice answer). Also, human labels a…

Agree, current "thinking" models are effectively "re-run this question N times, and determine the best answer", and this LLM-evaluating-LLM loop demonstrably leads to higher quality answers against objective metrics (in math, etc).

That’s… not how thinking models work. They tend to be iterative and serial, not parallel and then pick-one.

Re: AI agent benchmarks are broken

#66

Post by rf15 »

deepdarkforest wrote: post #4

It's very funny how many layers of abstraction we are going through. We have limited understanding of how LLM's work exactly and why. We now do post training with RL, which again, we don't have perfect understanding of it either. Then you stack LLMs calls and random tools, and you have agents, and you are attempting to benchmark those . (and this exclude voice, computer use agents etc). It's all just vibes,there is n…

> We have limited understanding of how LLM's work exactly and why.

blatantly untrue, and as a concept only useful to those who want to sell AI as this "magical thing" that "just works"

Re: AI agent benchmarks are broken

#67

Post by fragmede »

xnx wrote: post #33
qsort wrote: post #26

Earlier quoted context omitted.

We want machines that are better than humans, otherwise what purpose do they serve?

A machine with human level "AI" is still useful if it can run 24/7 and you can spin up 1M instances.

and they don't have family that gets sick or dies or come into work hungover or go off on political tangents and cause HR issues or want to take vacations or complain about bad working conditions.

Re: AI agent benchmarks are broken

#68

Post by brookst »

jacobr1 wrote: post #47
jstummbillig wrote: post #25

Earlier quoted context omitted.

> using a judge of the same architecture as the thing being judged maximizes the probability of fundamental failure of the benchmark to be valid due to the judge having the exact same blind spots as the thing under test. That's what humans do all the time. What's the fundamental difference? Or are you saying that's also broken?

The equivalent would be having the _same human_ review their own work. We require others with different experience and fresh eyes for secondary review and for the most important task multiple people. To some extent the same llm with a new context history and different prompt is sorta like that ... but still is much weaker than using a different system entirely.

How do you feel about o3 reviewing 4o-mini?

Re: AI agent benchmarks are broken

#69

Post by brookst »

rsynnott wrote: post #28
jstummbillig wrote: post #25

Earlier quoted context omitted.

> using a judge of the same architecture as the thing being judged maximizes the probability of fundamental failure of the benchmark to be valid due to the judge having the exact same blind spots as the thing under test. That's what humans do all the time. What's the fundamental difference? Or are you saying that's also broken?

... I mean, when evaluating "45 + 8 minutes" where the expected answer was "63 minutes", as in the article, a competent human reviewer does not go "hmm, yes, that seems plausible, it probably succeeded, give it the points". I know LLM evangelists love this "humans make mistakes too" line, but, really, only an _exceptionally_ incompetent human evaluator would fall for that one.

have you ever hired human evaluators at scale? They make all sorts of mistakes. Relatively low probability, so it’s a noise factor in, but I have yet to meet the human who is 100% accurate at simple tasks done thousands of times.

Re: AI agent benchmarks are broken

#70

Post by dmbche »

dmbche wrote: post #43
DonHopkins wrote: post #40

Earlier quoted context omitted.

It's like using steel to produce steel. What else are you going to use? Bamboo?

I'm not sure if I'm dense, but we don't use steel to make steel (whether crucibles or "feed material"). The first person to make steel made it without steel didn't they? Did I miss something? Edit0: fun tidbit - Wootz steel was made with crucibles of clay with rice husks mixed in (husks would carbonize quickly and introduce air layers to better isolate) and many seemingly random objects (fruits, vegetation) were adde…

Second fun tidbit : Bamboo was used as the fuel source in some furnaces - they did indeed use bamboo like the parent comment mentionned.
Post reply on HN