Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exploiting the most prominent AI agent benchmarks

rdi.berkeley.edu

Post reply on HN
Print view View original

11–20 of 175 posts

Re: Exploiting the most prominent AI agent benchmarks

#11

Post by siva7 »

operatingthetan wrote: post #5
ggillas wrote: post #2

This is a phenomenal paper on exploits and hopefully changes the way benchmarking is done. From the paper: We achieved near-perfect scores on all of them without solving a single task. The exploits range from the embarrassingly simple (sending {} to FieldWorkArena) to the technically involved (trojanizing binary wrappers in Terminal-Bench), but they all share a common thread: the evaluation was not designed to resist…

>hopefully changes the way benchmarking is done. Yeah the path forward is simple: check if the solutions actually contain solutions. If they contain exploits then that entire result is discarded.

Could it really be that not only we vibeslop all apps nowadays but also don't care to even check how ai solved a benchmark it claimed solved?

Re: Exploiting the most prominent AI agent benchmarks

#12

Post by lnrd »

I'm honestly confused by the design of SWE-bench and why is considered reliable.

It's based on existing GitHub PRs and Issues, the full dataset is on HuggingFace and is one year old now. All frontier models 100% have those issues and PRs in their training data so obviously they are good at reproducing fixes for them when confronted with the same codebase and similar requests. Am I missing something? How is this considered the most reliable benchmark?

Re: Exploiting the most prominent AI agent benchmarks

#13

Post by subulaz »

ModernMech wrote: post #7
charcircuit wrote: post #4

I always assumed that these benchmarks would happen in a sandbox. I'm surprised that no one realized this sooner.

I'm surprised anyone took them seriously in the first place.

a LOT of the people who love benchmarks are middle management hard-selling GenAI/LLM as magic tech sauce to vaguely technical executives who only want to know about the money aka headcount savings they so desperately desire.

their collective butts are already glued to the hype train as they chase numbers they (often) manufactured to justify the latest round of tech spend.

lots of good use cases out there - like the incredible progress with medical imaging analysis or complex system models for construction - and lots of crap use cases that need benchmarks to cosplay relevance.

Re: Exploiting the most prominent AI agent benchmarks

#15

Post by operatingthetan »

siva7 wrote: post #11
operatingthetan wrote: post #5

Earlier quoted context omitted.

>hopefully changes the way benchmarking is done. Yeah the path forward is simple: check if the solutions actually contain solutions. If they contain exploits then that entire result is discarded.

Could it really be that not only we vibeslop all apps nowadays but also don't care to even check how ai solved a benchmark it claimed solved?

Probably a more interesting benchmark is one that is scored based on the LLM finding exploits in the benchmark.

Re: Exploiting the most prominent AI agent benchmarks

#18

Post by ZeroGravitas »

operatingthetan wrote: post #5
ggillas wrote: post #2

This is a phenomenal paper on exploits and hopefully changes the way benchmarking is done. From the paper: We achieved near-perfect scores on all of them without solving a single task. The exploits range from the embarrassingly simple (sending {} to FieldWorkArena) to the technically involved (trojanizing binary wrappers in Terminal-Bench), but they all share a common thread: the evaluation was not designed to resist…

>hopefully changes the way benchmarking is done. Yeah the path forward is simple: check if the solutions actually contain solutions. If they contain exploits then that entire result is discarded.

In human multiple choice tests they sometimes use negative marking to discourage guessing. It feels like exploits should cancel out several correct solutions.

Re: Exploiting the most prominent AI agent benchmarks

#19

Post by SpicyLemonZest »

lnrd wrote: post #12

I'm honestly confused by the design of SWE-bench and why is considered reliable. It's based on existing GitHub PRs and Issues, the full dataset is on HuggingFace and is one year old now. All frontier models 100% have those issues and PRs in their training data so obviously they are good at reproducing fixes for them when confronted with the same codebase and similar requests. Am I missing something? How is this consi…

Frontier model developers do not consider SWE-bench to be reliable. OpenAI announced in February (https://openai.com/index/why-we-no-longer-evaluate-swe-bench...) that they consider it hopelessly contaminated, advocating for a new version SWE-bench Pro that was published more recently. (They seem to believe that even the publicly accessible part of the SWE-bench Pro problem set will be more resistant to training set contamination issues in the future, for reasons that to be honest I don't really understand.)
Post reply on HN