Arena AI Model ELO History • Hacker News Board

Re: Arena AI Model ELO History

#61

Post by refulgentis » Thu, May 14, 2026, 6:43 PM UTC

ninjalanternshk wrote: post #43

refulgentis wrote: post #10

Is this slop? It has wildly aggressive language that agrees with a subset of pop sentiment, re: models being “nerfed”. It promises to reveal this nerfing. Then, it goes on to…provide an innocuous mapping of LM Arena scores that always go up?

It links to the GitHub repo for the project, and while it’s not inconceivable that an AI bot would create and populate a functioning public GitHub repo, it’s pretty unlikely.

Whether there's a public GitHub repo is orthogonal to if the content on the site was written by AI, especially given it doesn't make any sense on it's own terms.

Orthogonal to that, 6 months ago, an AI making a repo was trivial. Wouldn't read it as a sign of anything.

Re: Arena AI Model ELO History

#62

Post by [deleted] » Thu, May 14, 2026, 11:03 PM UTC

retinaros wrote: post #60

tedsanders wrote: post #59

Earlier quoted context omitted.

What do you mean by this? We don’t train on evals, and if we did I’d quit on the spot. (The loose version of this that’s true is that there may exist eval data contamination in pretraining. This is a hard problem to fully solve.)

its not that loose of a version. its the reality and as probably is surely a focus of a dedicated post training RL-ing these kind of githubs. of course you would train specifically on the task. you would mix this eval data with others in thousands of githubs repos.

[deleted]

Re: Arena AI Model ELO History

#63

Post by jychang » Fri, May 15, 2026, 9:48 AM UTC

selcuka wrote: post #11

jychang wrote: post #9

Earlier quoted context omitted.

There's almost 0% chance that OpenAI doesn't quantize the model right off the bat. I am willing to bet large amounts of money that OpenAI would never release a model served as fully BF16 in the year of our lord 2026. That would be insane operationally. They're almost certainly doing QAT to FP4 for FFN, and a similar or slightly larger quant for attention tensors.

It's ok if they never release a BF16 model, but it's less ok if they release it, win the benchmarks, then quantise it after a few weeks.

That would be REALLY easy to detect. It'll be 4x slower.

The tokens/sec of the model is basically directly proportional of the memory bandwidth of the hardware it runs on. So either OpenAI has to gimp model performance for its entire life, or somehow magically speed it up 4x on the first day.