Live data from Hacker News

OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

ykilcher.com

11–20 of 64 posts

Re: OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

#11
post #4

This makes Turbo GPT 3.5 level AI free, private and finetuneable. OpenAI exclusivity shrinks now to GPT-4. That's why I don't think they will be able to keep a large market share in LLMs, any level of AI is going to get open and free soon. SOTA models are also easy to distill by API, very hard to defend against using chat logs as training data for other models. Once we all got one running, maybe in the OS, maybe in t…

The problem is that for-profit businesses like OpenAI have more money and compute than even millions of volunteers. I definitely believe we'll get an open GPT-4 eventually, but by then OpenAI will have GPT-5, and so on.

It's a shame really: the ultimate cause is the massive amount of wealth inequality we have today. If private entities and governments didn't have so much resources compared to individuals, I'm certain an open-source AI would be the biggest, because open-source has intrinsic benefits over closed-source: you have many people all working on the same project vs. multiple siloed groups, and anyone not affiliated with the private service is biased to use and support the open one. This is why the best operating systems, programming languages, and other software are all open-source: more money != better software, you don't need money to build software as much as you need intelligence and work ethic. But with AI, the #1 limiting factor is web-scraping required to get all of the data, and GPUs to train a model with it (maybe also money to pay Mechanical-Turk workers for simple classification; but perhaps enough volunteers could beat this, plus it seems like unskilled classification is becoming less important since the models can do this on their own).

That's not to say open-source AI won't be great, and I also think most places will use it. Especially if OpenAI is too expensive and/or disallows what they are trying to do. It does put pressure on OpenAI to be more lenient with pricing and acceptable use, and also to keep improving. But unless we address the massive wealth inequality, which is why LAION has substantially less funding than not just OpenAI but also some of the other startups, it's going to always lag behind.

Re: OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

#12
post #4

This makes Turbo GPT 3.5 level AI free, private and finetuneable. OpenAI exclusivity shrinks now to GPT-4. That's why I don't think they will be able to keep a large market share in LLMs, any level of AI is going to get open and free soon. SOTA models are also easy to distill by API, very hard to defend against using chat logs as training data for other models. Once we all got one running, maybe in the OS, maybe in t…

Do you have any evidence that this is GPT-3.5 level, or are you just repeating what they said? We have an abundance of claimed capabilities already; that's not what's lacking.

Re: OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

#13
This is the same Pythia and Llama based models right?

If so, they certainly aren't ChatGPT level in their quality. Impressive, potentially useful, but not ChatGPT.

Still an incredible effort, the RLHF data here might eventually make an Open Source ChatGPT possible, but these models are not that.

Re: OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

#14
It's awesome that the OpenAssistant project made it this far with a lot of crowed-sourced input. Congrats to the whole team that works really hard trying to create a truly open LLM.

One thing that puzzles me though, is that for the GPT-3.5 comparison, the model used is trained using both OpenAssistant and alpaca data, which is not free due to the OpenAI license used to generate the data. Isn't that defeating the purpose?

"... Completions were generated using pythia-12b-deduped fine-tuned on the OpenAssistant and Alpaca [9] dataset as well as gpt-3.5-turbo using the OpenAI API..."

Re: OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

#17
post #5
post #4

This makes Turbo GPT 3.5 level AI free, private and finetuneable. OpenAI exclusivity shrinks now to GPT-4. That's why I don't think they will be able to keep a large market share in LLMs, any level of AI is going to get open and free soon. SOTA models are also easy to distill by API, very hard to defend against using chat logs as training data for other models. Once we all got one running, maybe in the OS, maybe in t…

What beautiful garden? Are you completely ignoring the fact that OpenAI is made possible because it scraped the entire Web (the actual garden) and made a query index out of it? Do you not have any respect for people who actually spent their time and creativity to provide the information necessary for this model to even work? Ignorance is bliss I guess.

Out of curiosity, do you use ad blocking software?

Re: OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

#18
post #4

This makes Turbo GPT 3.5 level AI free, private and finetuneable. OpenAI exclusivity shrinks now to GPT-4. That's why I don't think they will be able to keep a large market share in LLMs, any level of AI is going to get open and free soon. SOTA models are also easy to distill by API, very hard to defend against using chat logs as training data for other models. Once we all got one running, maybe in the OS, maybe in t…

Side question, but how do these models are benchmarked, and how is this subfield evolving these days? I have seen many papers relying on standard student tests performance, but they don’t seem very accurate since LLAMA-based models perform almost as good as chatGPT (3/3.5) despite being apparently being an order of magnitude worse in practice.

Re: OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

#19
post #12
post #4

This makes Turbo GPT 3.5 level AI free, private and finetuneable. OpenAI exclusivity shrinks now to GPT-4. That's why I don't think they will be able to keep a large market share in LLMs, any level of AI is going to get open and free soon. SOTA models are also easy to distill by API, very hard to defend against using chat logs as training data for other models. Once we all got one running, maybe in the OS, maybe in t…

Do you have any evidence that this is GPT-3.5 level, or are you just repeating what they said? We have an abundance of claimed capabilities already; that's not what's lacking.

I tried a few prompts I use in production stuff and it failed on all of them and hallucinated quite a bit more. All of these models are optimized for the gimmicky chatbot stuff that seems impressive to a casual user, but not for comparable capabilities to GPT-3.5. I wish what the parent said was true because it would save me money!

Re: OpenAssistant Conversations – Democratizing Large Language Model Alignment [pdf]

#20
post #5
post #4

This makes Turbo GPT 3.5 level AI free, private and finetuneable. OpenAI exclusivity shrinks now to GPT-4. That's why I don't think they will be able to keep a large market share in LLMs, any level of AI is going to get open and free soon. SOTA models are also easy to distill by API, very hard to defend against using chat logs as training data for other models. Once we all got one running, maybe in the OS, maybe in t…

What beautiful garden? Are you completely ignoring the fact that OpenAI is made possible because it scraped the entire Web (the actual garden) and made a query index out of it? Do you not have any respect for people who actually spent their time and creativity to provide the information necessary for this model to even work? Ignorance is bliss I guess.

> Do you not have any respect for people who actually spent their time and creativity to provide the information necessary for this model to even work?

Yes, which is why I'm delighted to be able to filter out the advertizing spam that subhuman scum traffic alongside the outputs of creativity.

Post reply on HN