[deleted]
Ask HN: Can you crowdfund the compute for GPT?
151–157 of 157 posts
Re: Ask HN: Can you crowdfund the compute for GPT?
#152Earlier quoted context omitted.
oh wow, that was a surprisingly awesome story. thanks for sharing! this maaaay be covered in the Pile's writeup (which i have not yet read) but i wonder who was curating the overall "mix" of the content. seems easily biased to, say, public domain books, since the corpus is easily available. when people say things like "GPT3 has been trained on all of the internet" i suspect this is a gross exaggeration. In reality it…
Stella! She’s awesome. https://twitter.com/blancheminerva?s=21&t=Gt6YrATJHnmY046Mdz... Also bmk. https://twitter.com/nabla_theta?s=21&t=Gt6YrATJHnmY046MdzhYD... They did the legwork of writing the paper and getting everything into a presentable format. A bunch of other people helped too; I wasn’t as involved as I could’ve been. It was all discord-based. As far as I know it was the first serious research collaboration…
thank you for those names, ive added to my watchlist.
could be a good story to write up, i think the people who do the hard work behind all the datasets dont get any of the glory of the ML models that get built on top of it
Re: Ask HN: Can you crowdfund the compute for GPT?
#153Earlier quoted context omitted.
So is blockchain just an economic and architectural model to run crowdsourced things of value like GPT models? I mean it seems like a good first use case.
They usually aren’t calculating anything interesting, but in principal there could be something like that made.
Last time I saw a project aiming to be like a fast decentralized REPL but with blockchain.
Re: Ask HN: Can you crowdfund the compute for GPT?
#154Earlier quoted context omitted.
Data isn't the hard part here, plenty is available, even with all the necessary preprocessing.
A large amount of data is not an issue, but obtaining a high amount of high-quality data is challenging. This is why open-source models do not perform as well as GPT-3 models in real-world usage.
Sorry, since the Pile and C4 https://huggingface.co/datasets/c4 (and, more generally, common crawl) and BigCode https://www.bigcode-project.org/ became available, this argument ceased being the real moat.
The real moat is more about the lack of concentrated compute, ML engineering, and, more generally, prosaic lack of political with outside a few orgs.
Re: Ask HN: Can you crowdfund the compute for GPT?
#155I've often wondered why a service doesn't exist that allows you to rent out your graphics card for the large data processing needed for training models. Like mining bitcoin except you are doing something actually useful and getting paid actual money for it. Example: - Company Alpha needs $40,000,000 worth of cloud computing for their training model - Company Beta provides them said cloud computing for $30,000,000 fro…
Sheepit[0] does this with peoples Blender files, it’s awesome. I use it often and donate gpu time when I’m not using it.
Re: Ask HN: Can you crowdfund the compute for GPT?
#156I guess I may be behind the times on some of these AI efforts, but I'm guessing that the model involves a big black box blob of data.
I'd be interested to know how models evolve with additional training, and wonder if there are additive identities that would allow two separately trained models to be combined.
Re: Ask HN: Can you crowdfund the compute for GPT?
#157Earlier quoted context omitted.
Surprised no one has commented this but the latency requires the model to be trained in tiny fragments on each device which is currently a field of research that is being explored. As it stands now basically all of a model needs to be loaded into memory. There’s a whole field here and people exploring this problem, colloquially solving this would enable Federated Learning and whoever figures this out will far eclipse…
Where can I read more about this field of research?
This initiative is getting some pretty good practical results https://arxiv.org/abs/2209.01188, taking a look at their citations should lead you to other examples in the field