Gpt2-Chatbot Removed from Lmsys
lmsys.org
Gpt2-Chatbot Removed from Lmsys
1–10 of 14 posts
Re: Gpt2-Chatbot Removed from Lmsys
#2[deleted]
Re: Gpt2-Chatbot Removed from Lmsys
#3I don't understand the connection between the submission title and the actual page it links to. Am I missing something obvious?
Re: Gpt2-Chatbot Removed from Lmsys
#4[deleted]
Re: Gpt2-Chatbot Removed from Lmsys
#5I don't understand the connection between the submission title and the actual page it links to. Am I missing something obvious?
There was a big rumor that this Gpt2 chatbot was ChatGPT 4.5 or 5. That model is no longer on lmsys. See big discussion: https://news.ycombinator.com/item?id=40199715
Re: Gpt2-Chatbot Removed from Lmsys
#6I don't understand the connection between the submission title and the actual page it links to. Am I missing something obvious?
Sorry, should have clarified. The part of the site that states the model is currently unavailable links to their evaluation policies which is the link used for this submission :)
Re: Gpt2-Chatbot Removed from Lmsys
#7The news here is that gpt2-chatbot is confirmed to be an anonymized, unreleased model. See yesterday's discussion here: https://news.ycombinator.com/item?id=40199715 The https://chat.lmsys.org page now currently says this:
> *gpt2-chatbot is currently unavailable.* See our model evaluation policy [here].
That link takes you to https://lmsys.org/blog/2024-03-01-policy , which is this post's URL. The page was edited yesterday (when gpt2-chatbot was added) to include this section:
> Evaluating unreleased models: We collaborate with open-source and commercial model providers to bring their unreleased models to community for preview testing.
> Model providers can test their unreleased models anonymously, meaning the models' names will be anonymized. A model is considered unreleased if its weights are neither open, nor available via a public API or service. Evaluating an unreleased model consists of the following steps:
> 1. Add the model to Arena with an anonymous label. i.e., its identity will not be shown to users.
> 2. Keep it until we accumulate enough votes for its rating to stabilize or until the model provider withdraws it.
> 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details).
> 4. Remove the model from Arena.
Re: Gpt2-Chatbot Removed from Lmsys
#8The news here is that gpt2-chatbot is confirmed to be an anonymized, unreleased model. See yesterday's discussion here: https://news.ycombinator.com/item?id=40199715 The https://chat.lmsys.org page now currently says this: > *gpt2-chatbot is currently unavailable.* See our model evaluation policy [here]. That link takes you to https://lmsys.org/blog/2024-03-01-policy , which is this post's URL. The page was edited ye…
It's possible they temporarily removed it because it broke the whole anonymised part of that. It literally had information about itself in its system prompt and people could deduce it was running on an OAI API because of its API error response formatting. Either OAI collected all the data they need or it'll be back with some changes.
Re: Gpt2-Chatbot Removed from Lmsys
#9The news here is that gpt2-chatbot is confirmed to be an anonymized, unreleased model. See yesterday's discussion here: https://news.ycombinator.com/item?id=40199715 The https://chat.lmsys.org page now currently says this: > *gpt2-chatbot is currently unavailable.* See our model evaluation policy [here]. That link takes you to https://lmsys.org/blog/2024-03-01-policy , which is this post's URL. The page was edited ye…
It's possible they temporarily removed it because it broke the whole anonymised part of that. It literally had information about itself in its system prompt and people could deduce it was running on an OAI API because of its API error response formatting. Either OAI collected all the data they need or it'll be back with some changes.
They rate limited it at some point later in the day. The rate limiting applied to all the other OpenAI models also, but none of the non-OpenAI models. Just another data point of validation.
Re: Gpt2-Chatbot Removed from Lmsys
#10Earlier quoted context omitted.
It's possible they temporarily removed it because it broke the whole anonymised part of that. It literally had information about itself in its system prompt and people could deduce it was running on an OAI API because of its API error response formatting. Either OAI collected all the data they need or it'll be back with some changes.
They rate limited it at some point later in the day. The rate limiting applied to all the other OpenAI models also, but none of the non-OpenAI models. Just another data point of validation.
Yeah it's also possible they temporarily took it down because of the flood of users that were slowing down the whole site for everyone. Potential for it to be back but the rate limits are really low, something like 2 queries per user per day