Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings

#1

Post by MMMercy2 » Wed, May 03, 2023, 7:26 PM UTC

Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings
lmsys.org

Re: Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings

#2

Post by HoshinoAI » Wed, May 03, 2023, 8:14 PM UTC

It's glad to see the old technique is used for new models.

You may also learn from 7.33 dota update which uses a new ranking algorithm called Glicko.

Re: Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings

#3

Post by zhisbug » Wed, May 03, 2023, 8:19 PM UTC

HoshinoAI wrote: post #2

It's glad to see the old technique is used for new models. You may also learn from 7.33 dota update which uses a new ranking algorithm called Glicko.

could you provide any reference? Is it a variant of ELO?

Re: Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings

#4

Post by weichiang » Wed, May 03, 2023, 8:20 PM UTC

Surprised to learn StableLM is worse than plain LLaMA. link to their leaderboard: leaderboard.lmsys.org

Re: Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings

#5

Post by HoshinoAI » Wed, May 03, 2023, 8:36 PM UTC

zhisbug wrote: post #3

HoshinoAI wrote: post #2

It's glad to see the old technique is used for new models. You may also learn from 7.33 dota update which uses a new ranking algorithm called Glicko.

could you provide any reference? Is it a variant of ELO?

check matchmaking section: https://www.dota2.com/newfrontiers

Valve listed some reason for making the change.

https://en.wikipedia.org/wiki/Glicko_rating_system

Re: Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings

#6

Post by circuit10 » Wed, May 03, 2023, 8:50 PM UTC

weichiang wrote: post #4

Surprised to learn StableLM is worse than plain LLaMA. link to their leaderboard: leaderboard.lmsys.org

I’ve heard that it’s really bad for it’s size

Re: Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings

#7

Post by lee101 » Wed, May 03, 2023, 9:02 PM UTC

[dead]

Re: Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings

#8

Post by freediver » Thu, May 04, 2023, 5:37 AM UTC

This is a very good idea.

Re: Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings

#9

Post by aaron695 » Thu, May 04, 2023, 12:34 PM UTC

[dead]

Re: Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings

#10

Post by bradknowles » Mon, May 08, 2023, 5:33 AM UTC

zhisbug wrote: post #3

HoshinoAI wrote: post #2

It's glad to see the old technique is used for new models. You may also learn from 7.33 dota update which uses a new ranking algorithm called Glicko.

could you provide any reference? Is it a variant of ELO?

Elo != ELO.

One is a rating system named after the creator, Arpad Elo. See https://en.wikipedia.org/wiki/Elo_rating_system

The other is a rock band that was formed in 1970. See https://en.wikipedia.org/wiki/Electric_Light_Orchestra