The Leaderboard Illusion
arxiv.org
The Leaderboard Illusion
1–10 of 56 posts
Re: The Leaderboard Illusion
#2I think this is a really interesting paper from Cohere, it really feels that at this point in time you can't trust any public benchmark, and you really need your own private evals.
Re: The Leaderboard Illusion
#3The fact those big LLM developers devote a significant amount of effort to game benchmarks is a big show of confidence that they are making progress towards AGI and will recoup those billions of dollars and man-hours/s
Re: The Leaderboard Illusion
#4The fact those big LLM developers devote a significant amount of effort to game benchmarks is a big show of confidence that they are making progress towards AGI and will recoup those billions of dollars and man-hours/s
Are the benchmark prompts public and isn't that where the problem lies?
Re: The Leaderboard Illusion
#5[flagged]
Re: The Leaderboard Illusion
#6Sounds like classic inequality observed everywhere. Success leads to attention leads to more success.
Why spend evaluation resources on outsiders? Everyone wants to know who is exactly first second etc, after #10 it’s do your own evaluation if this is important to you.
Thus, we have this inequality.
Re: The Leaderboard Illusion
#7Sounds like classic inequality observed everywhere. Success leads to attention leads to more success. Why spend evaluation resources on outsiders? Everyone wants to know who is exactly first second etc, after #10 it’s do your own evaluation if this is important to you. Thus, we have this inequality.
Is it? Sounds to me like they run the same experiment many times and keep the "best" results. Which is cheating, or if the same thing is done in biomedical research: research fraud.
Re: The Leaderboard Illusion
#8Sounds like classic inequality observed everywhere. Success leads to attention leads to more success. Why spend evaluation resources on outsiders? Everyone wants to know who is exactly first second etc, after #10 it’s do your own evaluation if this is important to you. Thus, we have this inequality.
So attention is all you need?
Re: The Leaderboard Illusion
#9Sounds like classic inequality observed everywhere. Success leads to attention leads to more success. Why spend evaluation resources on outsiders? Everyone wants to know who is exactly first second etc, after #10 it’s do your own evaluation if this is important to you. Thus, we have this inequality.
So attention is all you need?
Bravo!
Re: The Leaderboard Illusion
#10Absolutely devastating for the credibility of FAIR.