OpenAI Vendor Lock-How OpenAI Went from Open Source to “Open Your Wallet”
1–6 of 6 posts
Re: OpenAI Vendor Lock-How OpenAI Went from Open Source to “Open Your Wallet”
#2How about GPT Neo? is it a valid alternative for some applications considering its much smaller training set?
What would it take to bring an open source model like Neo to OpenAI GPT levels in terms of resources around the world donating computing power?
How many people are we talking about and how long would it take? Is it even possible to coordinate something like that.
With so many crypto mining rigs left to rust around the planet one would think this could be a good cause to put them back to work.
Re: OpenAI Vendor Lock-How OpenAI Went from Open Source to “Open Your Wallet”
#3How about GPT Neo? is it a valid alternative for some applications considering its much smaller training set? What would it take to bring an open source model like Neo to OpenAI GPT levels in terms of resources around the world donating computing power? How many people are we talking about and how long would it take? Is it even possible to coordinate something like that. With so many crypto mining rigs left to rust a…
I'd like to know that as well. I went pretty far into looking at alternatives for the post. Best thing you can really do is try them yourself with a tool like this https://github.com/oobabooga/text-generation-webui
Benchmarks are one thing but I suspect if any were truly on par we would know about it.
Obviously it's partly compute cost, but I also suspect there's a lot of R&D that would need to be redone in the open.
How many tricks does openai's pretraining have that aren't found in some paper somewhere?
Re: OpenAI Vendor Lock-How OpenAI Went from Open Source to “Open Your Wallet”
#4How about GPT Neo? is it a valid alternative for some applications considering its much smaller training set? What would it take to bring an open source model like Neo to OpenAI GPT levels in terms of resources around the world donating computing power? How many people are we talking about and how long would it take? Is it even possible to coordinate something like that. With so many crypto mining rigs left to rust a…
> Is it even possible to coordinate something like that.
Hugging Face and other collaborators led the creation of BLOOM. Given how hot the competition is, I imagine some team will take advantage of this soon...
Re: OpenAI Vendor Lock-How OpenAI Went from Open Source to “Open Your Wallet”
#5How about GPT Neo? is it a valid alternative for some applications considering its much smaller training set? What would it take to bring an open source model like Neo to OpenAI GPT levels in terms of resources around the world donating computing power? How many people are we talking about and how long would it take? Is it even possible to coordinate something like that. With so many crypto mining rigs left to rust a…
I'd like to know that as well. I went pretty far into looking at alternatives for the post. Best thing you can really do is try them yourself with a tool like this https://github.com/oobabooga/text-generation-webui Benchmarks are one thing but I suspect if any were truly on par we would know about it. Obviously it's partly compute cost, but I also suspect there's a lot of R&D that would need to be redone in the open.…
Taking a paper and turning it into working production code is a non-trivial process, 100%.
Training big models takes a lot of random reads/writes and those tend to be pretty latency sensitive. There _may_ be a way to train this BitTorrent style with donated compute, but it's hard to say how many orders of magnitude slower that would be. (Do you need 2x more compute to do it distributes? 10? 100x?)
It is an interesting question to be able to explore this space more!
Re: OpenAI Vendor Lock-How OpenAI Went from Open Source to “Open Your Wallet”
#6It's a bit of a shame that open source LLMs lag so far behind OpenAI. But it's more to their credit: GPT-4 outclasses the competition from the published benchmarks.
Unless a team has been working on a competitor for a while, we're only going to see lacklustre fine-tunes of existing older models that can barely take on GPT-3.5 for the next few months.