Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta got caught gaming AI benchmarks

theverge.com

Post reply on HN
Print view View original

41–50 of 171 posts

Re: Meta got caught gaming AI benchmarks

#41

Post by genewitch »

light_hue_1 wrote: post #11
JKCalhoun wrote: post #5

Is LMArena junk now? I thought there was an aspect where you run two models on the same user-supplied query. Surely this can't be gamed? > “optimized for conversationality” I don't understand what that means - how it gives it an LMArena advantage.

LMArena was always junk. I work in this space and while the media takes it seriously most scientists don't. Random people ask random stuff and then it measures how good they feel. This is only a worthwhile evaluation if you're Google or Meta or OpenAI and you need to make a chartbot that keeps people coming back. It doesn't measure anything else useful.

I hear AI news from time to time from the M5M in the US - and the only place I've ever seen "LMArena" is on HN and in the LM studio discord. At a ratio of 5:1 at least.

Re: Meta got caught gaming AI benchmarks

#42

Post by ignoramous »

Ahmad al-Dahle, who leads "Gen AI" at Meta, wrote this on Twitter:

  ... We're also hearing some reports of mixed quality across different services ... 

  We've also heard claims that we trained on test sets -- that's simply not true and we would never do that. Our best understanding is that the variable quality people are seeing is due to needing to stabilize implementations.

  We believe the Llama 4 models are a significant advancement and we're looking forward to working with the community to unlock their value.
https://x.com/Ahmad_Al_Dahle/status/1909302532306092107 / https://archive.vn/JzONp

Re: Meta got caught gaming AI benchmarks

#43

Post by fullshark »

bko wrote: post #24
seydor wrote: post #12

tech companies competing over something that is losing them money is the most bizarre spectacle yet.

I think Meta sees AI and VR/AR as a platform. They got left behind on the mobile platform and forever have to contend with Apple semi-monopoly. They have no control and little influence over the ecosystem. It's an existential threat to them. They have vowed not to make that mistake again so are pushing for an open future that won't be dominated by a few companies that could arbitrarily hurt Meta's business. That's th…

Makes sense except for the fact that they leaked the llama weights by accident and needed to reverse engineer that explanation.

Re: Meta got caught gaming AI benchmarks

#45

Post by nightski »

genewitch wrote: post #36
nightski wrote: post #33

Earlier quoted context omitted.

Do you know that they made a bet on MoE? Meaning they abandonded dense models? I doubt that is the case. Just releasing MoE Llama 4 does not constitute a "bet" without further information. Also from what I can tell this performs better than models with parameter counts equal to one expert, and worse than fully dense models equal to total parameter count. Isn't that kind of what we'd expect? in what way is that a fail…

A 4x8 MOE performs better than an 8B but worse than a 32B, is your statement? My response would be, "so why bother with MOE?" However deepseek r1 is MOE from my understanding, but the "E" are all =>32B parameters. There's > 20 experts. I could be misinformed; however, even so, I'd say a MOE with 32B or even 70B experts will outperform (define this!) Models with equal parameter counts, because deepseek outperforms (de…

Easy, vastly improved inference performance on machines with larger RAM but lower bandwidth/compute. These are becoming more popular such as Apple's M series chips, AMD's strix halo series, and the upcoming DGX Spark from Nvidia.

Re: Meta got caught gaming AI benchmarks

#46

Post by throwaway25251 »

Mond_ wrote: post #9

The Llama 4 launch looks like a real debacle for Meta. The model doesn't look great. All the coverage I've seen has been negative. This is about what I expected, but it makes you wonder what they're going to do next. At this point it looks like they are falling behind the other open models, and made an ambitious bet on MoEs, without this paying off. Did Zuck push for the release? I'm sure they knew it wasn't ready ye…

I don't know about Llama 4. Competition is intense in this field so you can't expect everybody to be number 1. However, I think the performance culture at Meta is counterproductive. Incentives are misaligned, I hope leadership will try to improve it.

Employees are encouraged to ship half-baked features and move to another project. Quality isn't rewarded at all. The recent layoffs have made things even worse. Skilled people were fired, slowing down teams. I assume the goal was to push remaining employees to work even more, but I doubt this is working.

I haven't worked in enough companies of this size to be able to tell if alternatives are better, but it's very clear to me that Meta doesn't get the best from their employees.

Re: Meta got caught gaming AI benchmarks

#47

Post by jsheard »

bko wrote: post #24
seydor wrote: post #12

tech companies competing over something that is losing them money is the most bizarre spectacle yet.

I think Meta sees AI and VR/AR as a platform. They got left behind on the mobile platform and forever have to contend with Apple semi-monopoly. They have no control and little influence over the ecosystem. It's an existential threat to them. They have vowed not to make that mistake again so are pushing for an open future that won't be dominated by a few companies that could arbitrarily hurt Meta's business. That's th…

I wouldn't call what Meta is doing with VR/AR an "open future", it's pretty much the exact same playbook that Google and Apple used for their platforms. The only difference is Meta gets to be the landlord this time.

Re: Meta got caught gaming AI benchmarks

#48

Post by labrador »

Meta does themselves a disservice by having such a crappy public facing AI for people to try (meta.ai). I regularly use the web versions for GPT 4o, Deepseek, Grok, and Google Gemeni 2.5.

Meta is always the worst so I don't even bother anymore.

Re: Meta got caught gaming AI benchmarks

#49

Post by qoez »

Mond_ wrote: post #9

The Llama 4 launch looks like a real debacle for Meta. The model doesn't look great. All the coverage I've seen has been negative. This is about what I expected, but it makes you wonder what they're going to do next. At this point it looks like they are falling behind the other open models, and made an ambitious bet on MoEs, without this paying off. Did Zuck push for the release? I'm sure they knew it wasn't ready ye…

> it makes you wonder what they're going to do next

They're just gonna keep throwing money at it. This is a hobby and talent magnet for them, instagram is the money printer. They've been working on VR for like a decade with barely much results in terms of users (compared to costs). This will be no different.

Re: Meta got caught gaming AI benchmarks

#50

Post by zamadatix »

genewitch wrote: post #36
nightski wrote: post #33

Earlier quoted context omitted.

Do you know that they made a bet on MoE? Meaning they abandonded dense models? I doubt that is the case. Just releasing MoE Llama 4 does not constitute a "bet" without further information. Also from what I can tell this performs better than models with parameter counts equal to one expert, and worse than fully dense models equal to total parameter count. Isn't that kind of what we'd expect? in what way is that a fail…

A 4x8 MOE performs better than an 8B but worse than a 32B, is your statement? My response would be, "so why bother with MOE?" However deepseek r1 is MOE from my understanding, but the "E" are all =>32B parameters. There's > 20 experts. I could be misinformed; however, even so, I'd say a MOE with 32B or even 70B experts will outperform (define this!) Models with equal parameter counts, because deepseek outperforms (de…

DeepSeek V3/R1 are MoE with 256 experts per layer, actively using 1 shared expert and 8 routed experts per layer https://arxiv.org/html/2412.19437v1#S2:~:text=with%20MoE%20l... so you can't just take the active parameters and assume that's close to the size of a single expert (ignoring experts are per layer anyways and that there are still dense parameters to count).

Despite connotations of specialized intelligences the term "expert" provokes it's really mostly about scalability/efficiency of running large models. By splitting up sections of the layers and not activating all of them for each pass a single query takes less bandwidth, can be distributed across compute, and can be parallelized with other queries on the same nodes.

Post reply on HN