The logic for which models stay active when you click on a group of them is extremely not working
Arena AI Model ELO History
51–60 of 63 posts
Re: Arena AI Model ELO History
#52FYI, Elo isn't an acronym - it's a person's name. No need to capitalize it as ELO.
You're right, thanks for the heads up! Corrected (I can't edit the post on HN though).
Re: Arena AI Model ELO History
#53The logic for which models stay active when you click on a group of them is extremely not working
It'd be amazing if you could open an issue with a screenshot so I can take a look, I haven't been able to find issues when clicking on a group of models: https://github.com/mayerwin/AI-Arena-History/issues. Note: the model change points label being hidden when more than one curve is active is by design (to avoid cluttering), if this is what you were referring to.
Re: Arena AI Model ELO History
#54The Elo rating system measures relative performance to the other models. As the other models improve or rather newer better models enter the list, the Elo score of a given existing model will tend to decrease even though there might be no changes whatsoever to the model or its system prompt. You can't use Elo scores to measure decay of a models performance in absolute terms. For that you need a fixed harness running…
Yes, that is definitely a limitation. If all models become worse at the same pace, we won't see any degradation either. I couldn't find any historical dataset of model benchmarks (I'd really have loved that, to see how performance holds over time vs. the initial announcement), so the Elo data from Arena AI was the least imperfect proxy I could find.
Re: Arena AI Model ELO History
#55Is this slop? It has wildly aggressive language that agrees with a subset of pop sentiment, re: models being “nerfed”. It promises to reveal this nerfing. Then, it goes on to…provide an innocuous mapping of LM Arena scores that always go up?
[flagged]
Re: Arena AI Model ELO History
#56> the slow performance decays the decays are just more capable other models entering the population, making all prior models lose more frequently
No, that is not how ELO scores work.
It's a fitted Bradley Terry model, scaled to familiar Elo scores, anchored to wins against Mixtral-8x7B at 1114 (at least last time I looked at it). When you fit the model against historical data, and then you add another month of time that contains newer models, the relative strength of a given model might decline even if its absolute ability remained fixed.
Re: Arena AI Model ELO History
#57FYI, Elo isn't an acronym - it's a person's name. No need to capitalize it as ELO.
Don't bring me down…
Re: Arena AI Model ELO History
#58For what it's worth, I work at OpenAI and I can guarantee you that we don't switch to heavily quantized models or otherwise nerf them when we're under high load. It's true that the product experience can change over time - we're frequently tweaking ChatGPT & Codex with the intention of making them better - but we don't pull any nefarious time-of-day shenanigans or similar. You should get what you pay for.
> we don't switch to heavily quantized models That sounded like a press bulletin, so just to let you clarify yourself: Does that mean you may switch to lightly quantized models?
Thanks - let me clarify that we don’t switch to lightly quantized models by time of day or when under heavy load either.
(I used the adjective heavily because that’s what the original post said. I have no intention of making misleading but technically true statements.)
Re: Arena AI Model ELO History
#59Earlier quoted context omitted.
It's ok if they never release a BF16 model, but it's less ok if they release it, win the benchmarks, then quantise it after a few weeks.
that is for sure what everyone does. also they train on evals with the datasets that they would be bench against.
What do you mean by this? We don’t train on evals, and if we did I’d quit on the spot.
(The loose version of this that’s true is that there may exist eval data contamination in pretraining. This is a hard problem to fully solve.)
Re: Arena AI Model ELO History
#60Earlier quoted context omitted.
that is for sure what everyone does. also they train on evals with the datasets that they would be bench against.
What do you mean by this? We don’t train on evals, and if we did I’d quit on the spot. (The loose version of this that’s true is that there may exist eval data contamination in pretraining. This is a hard problem to fully solve.)
its not that loose of a version. its the reality and as probably is surely a focus of a dedicated post training RL-ing these kind of githubs. of course you would train specifically on the task. you would mix this eval data with others in thousands of githubs repos.