Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Study identifies weaknesses in how AI systems are evaluated

oii.ox.ac.uk

Post reply on HN
Print view View original

61–70 of 204 posts

Re: Study identifies weaknesses in how AI systems are evaluated

#61

Post by luckydata »

wolttam wrote: post #56

I'd like to see some video generation benchmarks. For example, one that tested a model's ability to generate POV footage of a humanoid form carrying out typical household tasks Even if it requires human evaluators at first, and even if the models completely suck at this task right now: it seems like the kind of task you'd want them to be good at, if you want these models to eventually carry out these tasks in embodie…

Generating footage wouldn't help with the opposite but navigating a simulation would which is a pretty standard type of evaluation for multimodal AIs designed to act in the real world.

Re: Study identifies weaknesses in how AI systems are evaluated

#62

Post by ACCount37 »

energy123 wrote: post #44
ACCount37 wrote: post #6

Earlier quoted context omitted.

Ratings on LMArena are too easily gamed. Even professional human evaluators are quite vulnerable to sycophancy and overconfident-and-wrong answers. And LMArena evaluators aren't professionals. A lot of the sycophancy mess that seeps from this generation of LLM stems from reckless tuning based on human feedback. Tuning for good LMArena performance has similar effects - and not at all by a coincidence.

It's biased to small context performance, which is why I don't pay much attention to it as a developer aside from a quick glance. I need performance at 40-100k tokens which models like Deepseek can't deliver but Gemini 2.5 Pro and ChatGPT 5.0 Thinking can.

And even "long term performance" splits itself into "performance on multi-turn instruction following" and "performance on agentic tasks" down the line. And "performance on agentic tasks" is a hydra in itself.

Capturing LLM performance with a single metric is a hopeless task. But even a single flawed metric beats no metrics at all.

Re: Study identifies weaknesses in how AI systems are evaluated

#63

Post by wolttam »

luckydata wrote: post #61
wolttam wrote: post #56

I'd like to see some video generation benchmarks. For example, one that tested a model's ability to generate POV footage of a humanoid form carrying out typical household tasks Even if it requires human evaluators at first, and even if the models completely suck at this task right now: it seems like the kind of task you'd want them to be good at, if you want these models to eventually carry out these tasks in embodie…

Generating footage wouldn't help with the opposite but navigating a simulation would which is a pretty standard type of evaluation for multimodal AIs designed to act in the real world.

Do you mean that it wouldn't help with ingesting footage and then determining how to act?

I can imagine a robotics architecture where you have one model generating footage (next frames for what it is currently seeing) and another dumber model which takes in the generated footage and only knows how to generate the motor/servo control outputs needed to control whatever robot platform it is integrated with.

I think that kind of architecture decoupling would be nice. It allows the model with all the world and task-specific knowledge to be agnostic from its underlying robot platform.

Re: Study identifies weaknesses in how AI systems are evaluated

#64

Post by 6510 »

bee_rider wrote: post #23

> "For example, if a benchmark reuses questions from a calculator-free exam such as AIME," the study says, "numbers in each problem will have been chosen to facilitate basic arithmetic. Testing only on these problems would not predict performance on larger numbers, where LLMs struggle." When models figure out how to exploit an effect that every clever college student does, that should count as a win. That’s a much mo…

I don't claim to know anything but I thought tool usage was a major sign of intelligence. For example floats are a wonderful technology but people use them as if chainsaws are great for cutting bread and butter. We now have entire languages that cant do basic arithmetic. I thought it was alarming: People it cant compute like this! Now we have language models, those are still computers, why cant we just give them.. you know... calculators? Arguably the best thing their universe has to offer.

edit: I forgot my point: calculating big numbers is not a real world problem anyone has.

Re: Study identifies weaknesses in how AI systems are evaluated

#65

Post by rrr_oh_man »

halJordan wrote: post #58
luke0016 wrote: post #39

Earlier quoted context omitted.

> reasoning ability should enable them to handle numbers of arbitrary size, just as it enables humans to do so, given some pencil and paper. Or given a calculator. Which it's running on. Which it in some sense is . There's something deeply ironic about the fact that we have an "AI" running on the most technologically advanced calculator in the history of mankind and...it can't do basic math.

This is a very unserious take. It's not ironic, because it's not a calculator.

What's meaning of `computer`, remind me quick?

Re: Study identifies weaknesses in how AI systems are evaluated

#66

Post by yunyu »

6510 wrote: post #64
bee_rider wrote: post #23

> "For example, if a benchmark reuses questions from a calculator-free exam such as AIME," the study says, "numbers in each problem will have been chosen to facilitate basic arithmetic. Testing only on these problems would not predict performance on larger numbers, where LLMs struggle." When models figure out how to exploit an effect that every clever college student does, that should count as a win. That’s a much mo…

I don't claim to know anything but I thought tool usage was a major sign of intelligence. For example floats are a wonderful technology but people use them as if chainsaws are great for cutting bread and butter. We now have entire languages that cant do basic arithmetic. I thought it was alarming: People it cant compute like this! Now we have language models, those are still computers, why cant we just give them.. yo…

We do? Tool use started coming in vogue around 2023

Re: Study identifies weaknesses in how AI systems are evaluated

#67

Post by lysace »

Tech companies/bloggers/press/etc are perpetually bad at benchmarks. For browsers they kept pushing simplistic javascript-centric benchmarks even when it was clear for at least 15 years that layout/paint/network/etc were the dominant bottlenecks in real-world usage.

It's primarily marketing-driven. I think the technical parts of companies need to attempt to own this more.

It gets really weird when engineering priorities shift because of these mostly irrelevant benchmarks.

Re: Study identifies weaknesses in how AI systems are evaluated

#68

Post by anamexis »

rrr_oh_man wrote: post #65
halJordan wrote: post #58

Earlier quoted context omitted.

This is a very unserious take. It's not ironic, because it's not a calculator.

What's meaning of `computer`, remind me quick?

Computer vision algorithms run on computers and they can’t do basic arithmetic.

My email client runs on my computer and it doesn’t do basic arithmetic either.

Something running on a computer does not imply that it can or should do basic arithmetic

Re: Study identifies weaknesses in how AI systems are evaluated

#69

Post by daveguy »

bbor wrote: post #55

I'm already quite put off by the title (it's science -- if you have a better benchmark, publish it!), but the contents aren't great either. It keeps citing numbers about "445 LLM benchmarks" without confirming whether any of the ones they deem insufficiently statistical are used by any of the major players. I've seen a lot of benchmarks, but maybe 20 are used regularly by large labs, max. "For example, if a benchmark…

The article references this review:

https://openreview.net/pdf?id=mdA5lVvNcU

And the review is pretty damning regarding statistical validity of LLM benchmarks.

Re: Study identifies weaknesses in how AI systems are evaluated

#70

Post by benjiro »

luke0016 wrote: post #39
layer8 wrote: post #25

Earlier quoted context omitted.

I don’t think the fact that LLMs can handle small numbers more reliably has anything to do with their reasoning ability. To the contrary, reasoning ability should enable them to handle numbers of arbitrary size, just as it enables humans to do so, given some pencil and paper. However: > Testing only on these problems would not predict performance on larger numbers, where LLMs struggle. Since performance on large numb…

> reasoning ability should enable them to handle numbers of arbitrary size, just as it enables humans to do so, given some pencil and paper. Or given a calculator. Which it's running on. Which it in some sense is . There's something deeply ironic about the fact that we have an "AI" running on the most technologically advanced calculator in the history of mankind and...it can't do basic math.

Thing is, a LLM is nothing but a prediction algorithm based upon what it trained. So it missing basic calculator functionality is a given. This is why tool usage is more and more a thing for LLMs. So that the LLM can from itself use a calculator for the actual math parts it needs. Thus increasing accuracy ...
Post reply on HN