Meta got caught gaming AI benchmarks

#1

Post by pseudolus » Tue, Apr 08, 2025, 11:29 AM UTC

Meta got caught gaming AI benchmarks
theverge.com

Re: Meta got caught gaming AI benchmarks

#2

Post by arresin » Tue, Apr 08, 2025, 2:05 PM UTC

I believe this was designed to flatter the prompter more / be more ingratiating. Which is a worry if true (what it says about the people doing the comparing).

Re: Meta got caught gaming AI benchmarks

#3

Post by deckar01 » Tue, Apr 08, 2025, 2:37 PM UTC

The top of that leaderboard is filled with closed weight experimental models.

Re: Meta got caught gaming AI benchmarks

#4

Post by etamponi » Tue, Apr 08, 2025, 3:28 PM UTC

Meta got caught _first_.

Re: Meta got caught gaming AI benchmarks

#5

Post by JKCalhoun » Tue, Apr 08, 2025, 3:50 PM UTC

Is LMArena junk now?

I thought there was an aspect where you run two models on the same user-supplied query. Surely this can't be gamed?

> “optimized for conversationality”

I don't understand what that means - how it gives it an LMArena advantage.

Re: Meta got caught gaming AI benchmarks

#6

Post by sumeno » Tue, Apr 08, 2025, 3:53 PM UTC

etamponi wrote: post #4

Meta got caught _first_.

Not even first, OpenAI got caught a while back

Re: Meta got caught gaming AI benchmarks

#7

Post by Mond_ » Tue, Apr 08, 2025, 3:56 PM UTC

sumeno wrote: post #6

etamponi wrote: post #4

Meta got caught _first_.

Not even first, OpenAI got caught a while back

Do you have a source for this? That's interesting (if true).

Re: Meta got caught gaming AI benchmarks

#8

Post by add-sub-mul-div » Tue, Apr 08, 2025, 3:58 PM UTC

arresin wrote: post #2

I believe this was designed to flatter the prompter more / be more ingratiating. Which is a worry if true (what it says about the people doing the comparing).

There's no end to the possible vectors of human manipulation with this "open-weight" black box.

Re: Meta got caught gaming AI benchmarks

#9

Post by Mond_ » Tue, Apr 08, 2025, 3:59 PM UTC

The Llama 4 launch looks like a real debacle for Meta. The model doesn't look great. All the coverage I've seen has been negative.

This is about what I expected, but it makes you wonder what they're going to do next. At this point it looks like they are falling behind the other open models, and made an ambitious bet on MoEs, without this paying off.

Did Zuck push for the release? I'm sure they knew it wasn't ready yet.

Re: Meta got caught gaming AI benchmarks

#10

Post by sumeno » Tue, Apr 08, 2025, 4:00 PM UTC

Mond_ wrote: post #7

sumeno wrote: post #6

Earlier quoted context omitted.

Not even first, OpenAI got caught a while back

Do you have a source for this? That's interesting (if true).

They got the dataset from Epoch AI for one of the benchmarks and pinky swore that they wouldn't train on it

https://techcrunch.com/2025/01/19/ai-benchmarking-organizati...