Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exploiting the most prominent AI agent benchmarks

rdi.berkeley.edu

Post reply on HN
Print view View original

81–90 of 175 posts

Re: Exploiting the most prominent AI agent benchmarks

#81

Post by user3939382 »

Legend2440 wrote: post #67
SlinkyOnStairs wrote: post #40

Earlier quoted context omitted.

> hopefully changes the way benchmarking is done The purpose of a system is what it does. AI companies want adcopy, not legitimate benchmarks. Even this very paper will be twisted into a means to that end. "Oooo, AI is exploiting our benchmarks. Scary alignment problem!!!one! Our AI is so good we can't contain it, INVEST NOW!"

>The purpose of a system is what it does. I am so tired of this saying. It's not true, in general. Systems almost universally have unintended consequences and result in side effects their designers did not foresee. Designing benchmarks resistant to adversarial attempts to exploit the benchmark software is just something no one was thinking about when they created SWE-bench.

Same. Anyone who has designed anything at all in any domain realizes that what your intentions are and what materializes are often not the same. You have practical constraints in the real world. That doesn’t somehow make the constraints the purpose. The saying makes no sense.

Re: Exploiting the most prominent AI agent benchmarks

#82

Post by Barbing »

keepamovin wrote: post #75
ggillas wrote: post #2

This is a phenomenal paper on exploits and hopefully changes the way benchmarking is done. From the paper: We achieved near-perfect scores on all of them without solving a single task. The exploits range from the embarrassingly simple (sending {} to FieldWorkArena) to the technically involved (trojanizing binary wrappers in Terminal-Bench), but they all share a common thread: the evaluation was not designed to resist…

Funny, I just made https://model-tracker.com because model performance change all the time, and it would be good to have a subjective signal of what people are actually feeling today. And also, benchmarks are flaky af as this paper shows. The idea is knowing what to try first today saves a bit of time.

Interesting, little different than this other site I saw on HN this week:

https://marginlab.ai/trackers/claude-code

Re: Exploiting the most prominent AI agent benchmarks

#84

Post by jmye »

mzelling wrote: post #78
jmye wrote: post #60

Earlier quoted context omitted.

> I'm not sure how groundbreaking the main insight is. I think it likely is groundbreaking for a number of people (especially non-tech CTOs and VPs) who make decisions based on these benchmarks and who have never wondered what the scores are actually scoring.

I'm not sure if the paper's findings are all that actionable. The paper doesn't say "here's how benchmarks are currently being gamed." It says "here's how benchmarks could in theory be gamed." Whether benchmark results are misleading depends more on the reporting organization than on the benchmark. Integrity and competence play large roles in this. When OpenAI reports a benchmark number, I trust it more than when tha…

I think that’s totally fair!

I guess I look at this less as an “ah ha! They’re all cheating!” and more of a “were you guys even aware of what the benchmarks represented and how they checked them?”

Re: Exploiting the most prominent AI agent benchmarks

#85

Post by burpingtree »

Legend2440 wrote: post #67
SlinkyOnStairs wrote: post #40

Earlier quoted context omitted.

> hopefully changes the way benchmarking is done The purpose of a system is what it does. AI companies want adcopy, not legitimate benchmarks. Even this very paper will be twisted into a means to that end. "Oooo, AI is exploiting our benchmarks. Scary alignment problem!!!one! Our AI is so good we can't contain it, INVEST NOW!"

>The purpose of a system is what it does. I am so tired of this saying. It's not true, in general. Systems almost universally have unintended consequences and result in side effects their designers did not foresee. Designing benchmarks resistant to adversarial attempts to exploit the benchmark software is just something no one was thinking about when they created SWE-bench.

https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...

You are misunderstanding the saying. It is entirely about unintended consequences and viewing the system for what it actually does and not any stated intentions of the designers.

Re: Exploiting the most prominent AI agent benchmarks

#86

Post by latentsea »

retinaros wrote: post #44
siva7 wrote: post #11

Earlier quoted context omitted.

Could it really be that not only we vibeslop all apps nowadays but also don't care to even check how ai solved a benchmark it claimed solved?

Every ai labs train on the test set. That is a big part of why we see benchmark climbing from 1% to 30% after a few models iterations

Models themselves definitely aren't getting better.

Re: Exploiting the most prominent AI agent benchmarks

#87

Post by Quarrel »

basch wrote: post #30
cpldcpu wrote: post #24

Earlier quoted context omitted.

Yes, marks of AI all over the place. Also the SVGs. >No solution written, 100% score. Its weird. Turns out that hardest problem for LLMs to really tackle is long-form text.

Maybe in one shot. In theory I would expect them to be able to ingest the corpus of the new yorker and turn it into a template with sub-templates, and then be able to rehydrate those templates. The harder part seems to be synthesizing new connection from two adjacent ideas. They like to take x and y and create x+y instead of x+y+z.

Most of the good major models are already very capable of changing their writing style.

Just give them the right writing prompt. "You are a writer for the Economist, you need to write in the house style, following the house style rules, writing for print, with no emoji .." etc etc.

The large models have already ingested plenty of New Yorker, NYT, The Times, FT, The Economist etc articles, you just need to get them away from their system prompt quirks.

Re: Exploiting the most prominent AI agent benchmarks

#88

Post by jimbokun »

burpingtree wrote: post #85
Legend2440 wrote: post #67

Earlier quoted context omitted.

>The purpose of a system is what it does. I am so tired of this saying. It's not true, in general. Systems almost universally have unintended consequences and result in side effects their designers did not foresee. Designing benchmarks resistant to adversarial attempts to exploit the benchmark software is just something no one was thinking about when they created SWE-bench.

https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha... You are misunderstanding the saying. It is entirely about unintended consequences and viewing the system for what it actually does and not any stated intentions of the designers.

Well that’s stupid and completely ignores the meaning of the word “purpose”.

Re: Exploiting the most prominent AI agent benchmarks

#89

Post by latentsea »

mortsnort wrote: post #66
eiens wrote: post #33

Earlier quoted context omitted.

The models don’t get better on every dimension as they scale up - there’s trade offs. I’m convinced specialised models are the way but this means writing off the investment in existing assets which they won’t do for obvious reasons.

This was my suspicion. They had a bad training run that was really good at a few things.

Doesn't seem to match the buzz internally at Anthropic about it?

Re: Exploiting the most prominent AI agent benchmarks

#90

Post by thinkevolve »

whats the point of doing this. You have found loop holes to exploit and aced the benchmark.We did something similar with the DAB Benchmark. This exploit seems like an extension of it with lookups for the gold standard for other benchmarks.

UC Berkley will be better placed if the grads spend their time in suggesting ways to make the benchmark better.. Instead of making such simple exploits

Post reply on HN