Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Study identifies weaknesses in how AI systems are evaluated

oii.ox.ac.uk

Post reply on HN
Print view View original

181–190 of 204 posts

Re: Study identifies weaknesses in how AI systems are evaluated

#181

Post by lielvilla »

I’m working a lot with TTS (Text-to-Speach), and it’s also a total wild west - even worse than LLMs in some ways. The demos are always perfect, but once you generate hundreds of minutes you start seeing volume drift, pacing changes, random artifacts, and occasional mispronunciations that never show up in the curated clips.

The big difference from LLMs is that we don’t really have production-grade, standardized benchmarks for long-form TTS. We need things like volume-stability across segments, speech-rate consistency, and pronunciation accuracy over a hard corpus.

I wrote up what this could look like here: https://lielvilla.com/blog/death-of-demo/

Re: Study identifies weaknesses in how AI systems are evaluated

#182

Post by scuff3d »

ACCount37 wrote: post #178
scuff3d wrote: post #156

Earlier quoted context omitted.

The big problem is that tech companies and journalist aren't transparent about this. They tout benchmark numbers constantly, like they're an object measure of capabilities.

That's because they are as close to "object measure capabilities" as anything we're ever going to get. Without benchmarks, you're down to evaluating model performance based on vibes and vibes only, which plain sucks. With benchmarks, you have numbers that correlate to capabilities somewhat.

So because there isn't a better measure it's okay that tech companies effectively lie and treat these benchmarks like they mean more then they actually do?

Re: Study identifies weaknesses in how AI systems are evaluated

#183

Post by nopurpose »

EdwardDiego wrote: post #180
jiggawatts wrote: post #146

Earlier quoted context omitted.

“Here’s the throughout at sustained 100% load with the same ten sample queries repeated over and over.” “The customers want lower latency at 30% load for unique queries.” “Err… we can scale up for more throughput!” ಠ_ಠ

And then when you ask if they disabled the query result cache before running their benchmarking, they blink and look confused.

Then you see 25% cache hit rate in production and realise that disabling it for benchmark is not a good option either.

Re: Study identifies weaknesses in how AI systems are evaluated

#184

Post by joegibbs »

nradov wrote: post #32
bee_rider wrote: post #23

> "For example, if a benchmark reuses questions from a calculator-free exam such as AIME," the study says, "numbers in each problem will have been chosen to facilitate basic arithmetic. Testing only on these problems would not predict performance on larger numbers, where LLMs struggle." When models figure out how to exploit an effect that every clever college student does, that should count as a win. That’s a much mo…

LLMs can probably be taught or configured to use external tools like Excel or Mathematica when such calculations are needed. Just like humans. There are plenty of untapped optimization opportunities.

I tried making a spreadsheet application and found that they’re not that great at working with 2D data, especially if there’s a lot of it. It’s harder to do search for a large spreadsheet than a large text file - you might get a range of thousands of numbers, how do you search that? And things like headers or important information may not be anywhere near where it’s focused which means it needs to read a ton of irrelevant context. For small sheets it works perfectly though, it’ll have to be something I’ll take another look at in the future.

Re: Study identifies weaknesses in how AI systems are evaluated

#185

Post by eucyclos »

sky2224 wrote: post #163
bjackman wrote: post #18

Earlier quoted context omitted.

For what it's worth, I work on platforms infra at a hyperscaler and benchmarks are a complete fucking joke in my field too lol. Ultimately we are measuring extremely measurable things that have an objective ground truth. And yet: - we completely fail at statistics (the MAJORITY of analysis is literally just "here's the delta in the mean of these two samples". If I ever do see people gesturing at actual proper analysi…

> we completely fail at statistics (the MAJORITY of analysis is literally just "here's the delta in the mean of these two samples". If I ever do see people gesturing at actual proper analysis, if prompted they'll always admit "yeah, well, we do come up with a p-value or a confidence interval, but we're pretty sure the way we calculate it is bullshit") Sort of tangential, but as someone currently taking an intro stati…

I found the same when I was taking intro to stats - I did get a much better intuition for what stuff meant after reading 'superforecasting' by tetlock and gardner - I find I'm recommending that book a lot come to think of it.

Re: Study identifies weaknesses in how AI systems are evaluated

#186

Post by jimmySixDOF »

bubblelicious wrote: post #4

I work on LLM benchmarks and human evals for a living in a research lab (as opposed to product). I can say: it’s pretty much the Wild West and a total disaster. No one really has a good solution, and researchers are also in a huge rush and don’t want to end up making their whole job benchmarking. Even if you could, and even if you have the right background you can do benchmarks full time and they still would be a mes…

Terminal Bench 2.0 just dropped and a big success factor they stress is the hand crafted phd level rollout tests they picked aprox 80 out of 120 with the incentive that anyone who contributed 3 would get listed as a paper author this resulted in high quality participation equivalent to foundation labs proprietary agentic RL data but it's FOSS.

Re: Study identifies weaknesses in how AI systems are evaluated

#187

Post by bjackman »

cyanydeez wrote: post #155
bjackman wrote: post #18

Earlier quoted context omitted.

For what it's worth, I work on platforms infra at a hyperscaler and benchmarks are a complete fucking joke in my field too lol. Ultimately we are measuring extremely measurable things that have an objective ground truth. And yet: - we completely fail at statistics (the MAJORITY of analysis is literally just "here's the delta in the mean of these two samples". If I ever do see people gesturing at actual proper analysi…

Id say your experience is being more monetized for growth for growth sake.

Actually I disagree that that's what's going on in the world of hyperscaler platforms. There is genuinely a staggering amount of money on the line with the efficiency of this platform. Plus, we have extremely sophisticated and performance-sensitive customers who are directly and continuously comparing us with our competitors.

This isn't just that nobody cares about the truth. People 100% care! If you actually degrade a performance metric as measured post-hoc in full prod, someone will 100% notice, and if you want to keep your feature un-rolled-back, you are probably gonna have to have a meeting with someone that has thousands of reports, and persuade them it's worth it to the business.

But you're always gonna have more luck if you can have that meeting _before_ you degrade it. But... it's usually pretty hard to figure out what the exact degradation is gonna be, because of the things in my previous comment...

Re: Study identifies weaknesses in how AI systems are evaluated

#188

Post by Dave_Wishengrad »

bubblelicious wrote: post #4

I work on LLM benchmarks and human evals for a living in a research lab (as opposed to product). I can say: it’s pretty much the Wild West and a total disaster. No one really has a good solution, and researchers are also in a huge rush and don’t want to end up making their whole job benchmarking. Even if you could, and even if you have the right background you can do benchmarks full time and they still would be a mes…

[dead]

Re: Study identifies weaknesses in how AI systems are evaluated

#189

Post by 3abiton »

calpaterson wrote: post #3

Definitely one of the weaker areas in the current LLM boom. Comparing models, or even different versions of the same model, is a pseudo-scientific mess. I'm still using https://lmarena.ai/leaderboard . Perhaps there is something better and someone will pipe up to tell me about it. But we use LLMs at work and have unexplainable variations between them. And when we get a prompt working reliably on one model, we often h…

> Comparing models, or even different versions of the same model, is a pseudo-scientific mess.

Reminder that in most cases, it's impossible to know if there is cross-contamination from the test set of the public benchmarks, as most LLMs are not truely open-source. We can't replicate them. So arguably it's worse in some cases, pretty much fraud if you account for the VC money pouring in. This is even more evident in unknown models from lesser known institutes like from UAE.

Re: Study identifies weaknesses in how AI systems are evaluated

#190

Post by [deleted] »

bubblelicious wrote: post #4

I work on LLM benchmarks and human evals for a living in a research lab (as opposed to product). I can say: it’s pretty much the Wild West and a total disaster. No one really has a good solution, and researchers are also in a huge rush and don’t want to end up making their whole job benchmarking. Even if you could, and even if you have the right background you can do benchmarks full time and they still would be a mes…

[deleted]
Post reply on HN