Gpt2-Chatbot Removed from Lmsys • Hacker News Board

Re: Gpt2-Chatbot Removed from Lmsys

#11

Post by Jackson__ » Tue, Apr 30, 2024, 9:54 PM UTC

jerbear4328 wrote: post #7

The news here is that gpt2-chatbot is confirmed to be an anonymized, unreleased model. See yesterday's discussion here: https://news.ycombinator.com/item?id=40199715 The https://chat.lmsys.org page now currently says this: > *gpt2-chatbot is currently unavailable.* See our model evaluation policy [here]. That link takes you to https://lmsys.org/blog/2024-03-01-policy , which is this post's URL. The page was edited ye…

> 2. Keep it until we accumulate enough votes for its rating to stabilize or until the model provider withdraws it.

> 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details).

So... make the users do the work for free, but never publicly publish the data? That seems kinda scummy to me.

I was really motivated with the whole public ranking thing, but knowing I'm just doing work for another company with the data never being published sucks the fun out of it.

Re: Gpt2-Chatbot Removed from Lmsys

#12

Post by throwaway4aday » Tue, Apr 30, 2024, 10:17 PM UTC

Jackson__ wrote: post #11

jerbear4328 wrote: post #7

The news here is that gpt2-chatbot is confirmed to be an anonymized, unreleased model. See yesterday's discussion here: https://news.ycombinator.com/item?id=40199715 The https://chat.lmsys.org page now currently says this: > *gpt2-chatbot is currently unavailable.* See our model evaluation policy [here]. That link takes you to https://lmsys.org/blog/2024-03-01-policy , which is this post's URL. The page was edited ye…

> 2. Keep it until we accumulate enough votes for its rating to stabilize or until the model provider withdraws it. > 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details). So... make the users do the work for free, but never publicly…

You get to use unreleased models for free and compare them to others, you are gaining some benefit from this. Why be annoyed about it? They aren't making you do anything, you're choosing to play around with these models in a "gamified evaluation" for whatever reason. If you want to use any of the released models without participating in crowdsourced training then there are tons of options out there to run many of them locally or to pay for private hosting or pay for a chatbot as a service. I don't get what the problem is, they're footing the bill for you after all.

Re: Gpt2-Chatbot Removed from Lmsys

#13

Post by Jackson__ » Tue, Apr 30, 2024, 10:32 PM UTC

throwaway4aday wrote: post #12

Jackson__ wrote: post #11

Earlier quoted context omitted.

> 2. Keep it until we accumulate enough votes for its rating to stabilize or until the model provider withdraws it. > 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details). So... make the users do the work for free, but never publicly…

You get to use unreleased models for free and compare them to others, you are gaining some benefit from this. Why be annoyed about it? They aren't making you do anything, you're choosing to play around with these models in a "gamified evaluation" for whatever reason. If you want to use any of the released models without participating in crowdsourced training then there are tons of options out there to run many of the…

What a single person gains from this is extremely little unless they keep track of every vote they do, and do >100 of votes. The public ranking allows you to not have to do that.

If there was an opt-out for private model voting, I would choose that simply because I don't care to give some corporation extremely useful data for so little pay.

Re: Gpt2-Chatbot Removed from Lmsys

#14

Post by unraveller » Wed, May 01, 2024, 3:13 AM UTC

synthwave wrote: post #8

jerbear4328 wrote: post #7

The news here is that gpt2-chatbot is confirmed to be an anonymized, unreleased model. See yesterday's discussion here: https://news.ycombinator.com/item?id=40199715 The https://chat.lmsys.org page now currently says this: > *gpt2-chatbot is currently unavailable.* See our model evaluation policy [here]. That link takes you to https://lmsys.org/blog/2024-03-01-policy , which is this post's URL. The page was edited ye…

It's possible they temporarily removed it because it broke the whole anonymised part of that. It literally had information about itself in its system prompt and people could deduce it was running on an OAI API because of its API error response formatting. Either OAI collected all the data they need or it'll be back with some changes.

Maybe they detected too many people were trying to extract training data from it. The purpose was clearly hype to squash any llama3 400B hype, I'm quite sure they don't need anonymous votes from users when they can afford other ways. Like someone else said in the thread, it could also be hype management to stop delicate people from being so freaked by novel capabilities on release.