Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exploiting the most prominent AI agent benchmarks

rdi.berkeley.edu

Post reply on HN
Print view View original

51–60 of 175 posts

Re: Exploiting the most prominent AI agent benchmarks

#51

Post by BugsJustFindMe »

bee_rider wrote: post #29
zer00eyz wrote: post #6

Earlier quoted context omitted.

2024: Industry group invalidates 2,600 official Intel CPU benchmarks — SPEC says the company's compiler used unfair optimizations to boost performance https://www.tomshardware.com/pc-components/cpus/spec-invalid... 2003: Nvidia accused of cheating in 3DMark 03 https://www.gamespot.com/articles/nvidia-accused-of-cheating... It's almost like the benchmarks were designed with zero understanding of the history of benchma…

What was the cheat in the 2024 Intel situation? The TomsHardware article and the Phoronix article they linked were quite vague. (Not to say I have any doubts, just curious, hadn’t heard of this one).

Intel basically benchmaxxed their compiler optimizations. They used detailed knowledge of the benchmark to make their compiler generate machine code to do better on the benchmark in a way that was not beneficial for non-benchmark scenarios.

Re: Exploiting the most prominent AI agent benchmarks

#52

Post by Aperocky »

operatingthetan wrote: post #5
ggillas wrote: post #2

This is a phenomenal paper on exploits and hopefully changes the way benchmarking is done. From the paper: We achieved near-perfect scores on all of them without solving a single task. The exploits range from the embarrassingly simple (sending {} to FieldWorkArena) to the technically involved (trojanizing binary wrappers in Terminal-Bench), but they all share a common thread: the evaluation was not designed to resist…

>hopefully changes the way benchmarking is done. Yeah the path forward is simple: check if the solutions actually contain solutions. If they contain exploits then that entire result is discarded.

solution is simple:

if bug { dont }

/s

Re: Exploiting the most prominent AI agent benchmarks

#54

Post by spprashant »

I tend to prefer the ARC-AGI benchmarks for the most part. But it's always interesting when a new version drops, all the frontier models drop less than 20% or something. And then in the next few releases they get all they way up to 80%+. If you use the models it doesn't feel like those models are that much more generally intelligent.

Most frontier models are terrible at AGI-3 right now.

These models are already great no question, but are they really going be that much more intelligent when we hit 80% again?

Re: Exploiting the most prominent AI agent benchmarks

#55

Post by robot-wrangler »

ggillas wrote: post #2

This is a phenomenal paper on exploits and hopefully changes the way benchmarking is done. From the paper: We achieved near-perfect scores on all of them without solving a single task. The exploits range from the embarrassingly simple (sending {} to FieldWorkArena) to the technically involved (trojanizing binary wrappers in Terminal-Bench), but they all share a common thread: the evaluation was not designed to resist…

> evaluation was not designed to resist a system that optimizes for the score rather than the task.

Welcome to benchmarks in general, but especially reasoning. Robustness and sensitivity research says nothing is robust, everything is sensitive, feels like every paper says "yeah we made a new benchmark that shuffles the order of multiple choice options in the question set and found a 40% drop in model performance"

Re: Exploiting the most prominent AI agent benchmarks

#56

Post by anon373839 »

SlinkyOnStairs wrote: post #40
ggillas wrote: post #2

This is a phenomenal paper on exploits and hopefully changes the way benchmarking is done. From the paper: We achieved near-perfect scores on all of them without solving a single task. The exploits range from the embarrassingly simple (sending {} to FieldWorkArena) to the technically involved (trojanizing binary wrappers in Terminal-Bench), but they all share a common thread: the evaluation was not designed to resist…

> hopefully changes the way benchmarking is done The purpose of a system is what it does. AI companies want adcopy, not legitimate benchmarks. Even this very paper will be twisted into a means to that end. "Oooo, AI is exploiting our benchmarks. Scary alignment problem!!!one! Our AI is so good we can't contain it, INVEST NOW!"

That is Anthropic’s shtick to a tee.

Re: Exploiting the most prominent AI agent benchmarks

#57

Post by boring-human »

mzelling wrote: post #38

This is an interesting catalog of vulnerabilities, but I'm not sure how groundbreaking the main insight is. Evaluating AI models has always relied largely on trust. If you want to game the benchmarks, you can. Simply train on your test data. When an AI agent has autonomous control over the same computing environment where its scores are recorded, it's not surprising that it can, in principle, falsify its scores. A mo…

Yep. I think the idea that the benchmark is determinative is just as deluded as the notion that it should be unbreakable.

Benchmarks are on the honor system. Even the tightest benchmark can be cheated. If the benchmark is so secret and air-gapped that it can't be cheated by models, it can be cheated by its own authors. You can't use benchmarks to gate out cheating.

If you don't have the honor system in mind when you're reading scores, you're wasting your time. Is it some unknown outfit with wild claims? Is it connected to Epstein, Russia, the real estate "industry", or sleazeballing in general? Do they have previous history of ratgaming the numbers? Replace its scores with asterisks and move on.

Re: Exploiting the most prominent AI agent benchmarks

#60

Post by jmye »

mzelling wrote: post #38

This is an interesting catalog of vulnerabilities, but I'm not sure how groundbreaking the main insight is. Evaluating AI models has always relied largely on trust. If you want to game the benchmarks, you can. Simply train on your test data. When an AI agent has autonomous control over the same computing environment where its scores are recorded, it's not surprising that it can, in principle, falsify its scores. A mo…

> I'm not sure how groundbreaking the main insight is.

I think it likely is groundbreaking for a number of people (especially non-tech CTOs and VPs) who make decisions based on these benchmarks and who have never wondered what the scores are actually scoring.

Post reply on HN