Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Can you crowdfund the compute for GPT?

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 157 posts

Re: Ask HN: Can you crowdfund the compute for GPT?

#141

Post by sillysaurusx »

swyx wrote: post #140
sillysaurusx wrote: post #138

Earlier quoted context omitted.

Not at all, I love talking about it. I was convinced that a model needed to be able to read like we do. And what do we do when we read? Pick up a book. That turns out to be surprisingly hard, at least for training data. Step one is to acquire the books. Step two is to turn them into a readable format for computers. Both steps were very hard. I lucked out on step one because The Eye happened to host all of bibliotok,…

oh wow, that was a surprisingly awesome story. thanks for sharing! this maaaay be covered in the Pile's writeup (which i have not yet read) but i wonder who was curating the overall "mix" of the content. seems easily biased to, say, public domain books, since the corpus is easily available. when people say things like "GPT3 has been trained on all of the internet" i suspect this is a gross exaggeration. In reality it…

Stella! She’s awesome. https://twitter.com/blancheminerva?s=21&t=Gt6YrATJHnmY046Mdz...

Also bmk. https://twitter.com/nabla_theta?s=21&t=Gt6YrATJHnmY046MdzhYD...

They did the legwork of writing the paper and getting everything into a presentable format. A bunch of other people helped too; I wasn’t as involved as I could’ve been.

It was all discord-based. As far as I know it was the first serious research collaboration to happen solely via chatroom.

bmk also got the 50GB of code from GitHub, I think. So that’s where GPT-J’s coding ability likely came from.

Re: Ask HN: Can you crowdfund the compute for GPT?

#142

Post by alexmolas »

Aromasin wrote: post #105
cuuupid wrote: post #39

Earlier quoted context omitted.

Surprised no one has commented this but the latency requires the model to be trained in tiny fragments on each device which is currently a field of research that is being explored. As it stands now basically all of a model needs to be loaded into memory. There’s a whole field here and people exploring this problem, colloquially solving this would enable Federated Learning and whoever figures this out will far eclipse…

To piggy back this comment, Federated Learning actually has a lot of other uses beyond just crowdsourcing. It has benefits to data privacy also. The medical industry has struggled sharing their data with other institutions without compromising their information. They're using it to decentralize the model making process. Intel is doing some work with Penn on the subject now, if people want to read further: https://www…

AFAIK Federated Learning is not a magic solution. This post explains how you can exploit FL if there's some untrusted server (http://www.cleverhans.io/2022/04/17/fl-privacy.html)

Re: Ask HN: Can you crowdfund the compute for GPT?

#143

Post by kmeisthax »

rysertio wrote: post #65
alexb_ wrote: post #25

I've often wondered why a service doesn't exist that allows you to rent out your graphics card for the large data processing needed for training models. Like mining bitcoin except you are doing something actually useful and getting paid actual money for it. Example: - Company Alpha needs $40,000,000 worth of cloud computing for their training model - Company Beta provides them said cloud computing for $30,000,000 fro…

There's a lot of problems. 1. How can I confirm that you've done the computation? 2. Privacy and security issues. Can I trust you too process my sensitive information? 3. Availability: is there a guarranty you won't just do half of it and then be on a hiatus for 2 months. But for everywhere these problems are solved we have decentralized cloud computing. For others you need to solve these problems.

SETI@home and Folding@home had to deal with these problems decades ago - even with a closed-source client people would mod it in questionable ways to cheat the leaderboards.

Any computation can be verified as having been done by, at a minimum, checking for reproducibility. This requires having each work unit be done twice and only issuing credit if both units match. For deep-learning applications "match" is relative: different compute accelerators are going to give different results. So, instead we can insist that all the floating-point outputs on the model have to match up to the first n bits of mantissa. Neural networks are actually really insensitive to small perturbations in their weights, and it's common to train on 16-bit floats to save time.

We can also exploit the loss function itself as a verification mechanism. Generally speaking training is more compute-heavy than inference[0], so we can just run the updated model on the training set and confirm that your trained model is better than the original you were provided with to start from. This will need upper bounds, too - if only to catch people trying to overfit the model to guarantee they get credit.

As for privacy and security... the answer is to not train on private data or things that people do not want to be trained. Period. This isn't even a problem solely with distributed computing. All AI training should be limited to either data provided with consent, or data that's so old that training on it would not cause harm.

Availability is a problem, but not necessarily one that most distributed computing projects actually have to deal with. There is a minor incentive to participate with the credit system; there's a leaderboard for the fastest/highest credit users and teams. And people do compete for those leaderboard slots, because that's effectively ad space.

[0] Model execution.

Re: Ask HN: Can you crowdfund the compute for GPT?

#144

Post by haldujai »

What is your desired objective and what are you going to train on? There are plenty of publicly available model checkpoints so you don’t have to start from scratch with large GPU clusters. There’s no point in repeating the pretraining that has already been done.

I’m not sure that compute is still the bottleneck right now, it’s fairly cheap to train LLMs. Many optimizations like DeepSpeed dramatically reduce computation requirements / increase throughout (I.e. 13b model on a single GPU).

If we’ve learned anything from the massive LLMs like PaLM it’s that scaling autoregressive models to infinity has diminishing returns and is not feasible to implement in inference infra. Google themselves acknowledge this resource limitation in the Med-PaLM paper when they discuss fine tuning a 450b parameter model.

We’re really in more of a dataset and training task era of AI/NLP gains. Scaling masked the issues of poor quality training data and conventional language modelling up to a certain point but we’re starting to see the problems with that (hallucination) in all of the big (Galactica is an example of the limitations in this).

OpenAI’s main advantage is that they paid humans to build a large labelled dataset for their RL objective. They’re offering ChatGPT for free to get more training data.

Re: Ask HN: Can you crowdfund the compute for GPT?

#145

Post by lossolo »

sinenomine wrote: post #121
beernet wrote: post #12

It's not that easy. Access to enough compute is one thing. However, you also need a proper dataset (beyond Common Crawl and Wikipedia), excellent research expertise and engineering capabilities. So even if you throw money or free credits for cloud compute out there it will not be enough. We've seen this happen with EleutherAI who were not capable of reaching their initial target of "replicating" GPT-3 and could only…

Data isn't the hard part here, plenty is available, even with all the necessary preprocessing.

A large amount of data is not an issue, but obtaining a high amount of high-quality data is challenging. This is why open-source models do not perform as well as GPT-3 models in real-world usage.

Re: Ask HN: Can you crowdfund the compute for GPT?

#146

Post by generalizations »

asim wrote: post #122
generalizations wrote: post #118

Earlier quoted context omitted.

> Separately, examination of religious and historic tests in alternate languages providing superior transliteration and translation into english than what humans could achieve. I'd be interested in something similar to this as well - but I think part of the problem is that depending on the language, the text might only be available as digital scans without OCR. In that case, given that it would be in difficult-to-OCR…

Write a model to do image to text? In the case of things like the Quran pretty much all of it is digitized already so consumable by a model. Other things could be figured out over time. I think for a lot of things this could lead to huge breakthroughs if done right.

I was thinking of ancient Arabic and Chinese texts that are obscure enough that no one's bothered to translate them yet.

Re: Ask HN: Can you crowdfund the compute for GPT?

#147

Post by theamk »

welshwelsh wrote: post #55
alexb_ wrote: post #25

I've often wondered why a service doesn't exist that allows you to rent out your graphics card for the large data processing needed for training models. Like mining bitcoin except you are doing something actually useful and getting paid actual money for it. Example: - Company Alpha needs $40,000,000 worth of cloud computing for their training model - Company Beta provides them said cloud computing for $30,000,000 fro…

>Like mining Bitcoin except you are doing something actually useful and getting paid actual money for it A business paying USD is never going to be competitive with a decentralized crypto compute market. Better to build something just like mining BTC, except you do useful work, and are paid in a cryptocurrency you can exchange for USD. Then businesses can build on top of that to make it more user friendly. The Golem…

I don't think businesses would be able to ever trust decentralized compute market. Where is the guarantee that the data/code that is sent to the network would not be intercepted and analyzed?

It's much safer to sign a contract with a real organization, especially the one which has a reputation to uphold. If someone like DigitalOcean steals data from servers, they can be sued, and there will be penalties. If some decentralized miner somewhere does that? Nothing you can do.

This only leaves decentralized compute to the applications where neither the data nor code does not matter. While I am sure there are applications like those (probably related to open-source or cryptocurrencies) I doubt they'll bring much money.

(And don't say "homomorphic encryption" -- the overhead there is so high it is much cheaper to just get a centralized server)

Re: Ask HN: Can you crowdfund the compute for GPT?

#148

Post by alasdair_ »

bobkazamakis wrote: post #123
alasdair_ wrote: post #106

Earlier quoted context omitted.

>1. How can I confirm that you've done the computation? The same problem applies to things like Mechanical Turk and other croudsourcing. The way I've dealt with the issue in the past is to start with zero trust and to have them do computations that I already know the answer to. After that, they do computations that are matched with a random other participant (the two results should match, if they don't, compare again…

> have them do computations that I already know the answer to. By definition, this is not having them do any computation. The proper solution at this time would be some trapdoor function that is easily verifiable (proof of work), at least while P != NP

>By definition, this is not having them do any computation.

If I ask you to compute the first million digits of pi to the power of 1.23456 and I already know the answer to validate it, how is this "by definition" not computation?

Re: Ask HN: Can you crowdfund the compute for GPT?

#150

Post by sfmike »

cuuupid wrote: post #39
alexb_ wrote: post #25

I've often wondered why a service doesn't exist that allows you to rent out your graphics card for the large data processing needed for training models. Like mining bitcoin except you are doing something actually useful and getting paid actual money for it. Example: - Company Alpha needs $40,000,000 worth of cloud computing for their training model - Company Beta provides them said cloud computing for $30,000,000 fro…

Surprised no one has commented this but the latency requires the model to be trained in tiny fragments on each device which is currently a field of research that is being explored. As it stands now basically all of a model needs to be loaded into memory. There’s a whole field here and people exploring this problem, colloquially solving this would enable Federated Learning and whoever figures this out will far eclipse…

bitorrent for GPUs?
Post reply on HN