81–90 of 274 posts
Earlier quoted context omitted.
Wonder if someone would be willing to start an open source project where we could crowdsource donations for training, and people could possibly donate their GPU usage for it.
That’s what Stability AI has been doing… There are already open source LLMs the size of GPT-3 such as OPT and Bloom
Out of curiosity, why aren't we crowd sourcing distributed training of LLMs where anyone can join by bringing their hardware or data? Moreover find a way to incorporate this into a blockchain so there is full transparency but also add in differential privacy to protect every participant. Am I being too crazy here?
There is also EleutherAI (https://www.eleuther.ai/about/) with GPT-NeoX (https://github.com/EleutherAI/gpt-neox).
Out of curiosity, why aren't we crowd sourcing distributed training of LLMs where anyone can join by bringing their hardware or data? Moreover find a way to incorporate this into a blockchain so there is full transparency but also add in differential privacy to protect every participant. Am I being too crazy here?
Earlier quoted context omitted.
Why would you want to retrain it from scratch every day? Stable Diffusion doesn't do that either.
Because things happen every day. If ChatGPT wants to compete with Google, staying up to date with recent events is the minimum bar.
Earlier quoted context omitted.
Why would you want to retrain it from scratch every day? Stable Diffusion doesn't do that either.
Because things happen every day. If ChatGPT wants to compete with Google, staying up to date with recent events is the minimum bar.
I wrote about one way to implement that pattern here: https://simonwillison.net/2023/Jan/13/semantic-search-answer...
Earlier quoted context omitted.
You can do it once, but probably not every day.
Why would you want to retrain it from scratch every day? Stable Diffusion doesn't do that either.
It's really interesting that these models are written in Python. Anyone know how much of a speed up using a faster language here would have? Maybe it's already off-loading a lot of the computation to C (I know many Python libraries do this), but I'd love to know.
1. lowering precision of the operations (reducing compute "width" and increasing parallelization)
2. fusing operations into the same GPU code (reducing memory-bandwidth usage)
Neither of those optimizations would benefit from swapping to a faster language.
Why? The typical "large" neural network operation runs on the order of a dozen microseconds to milliseconds. Models are usually composed of hundred if not thousands of these. The overhead of using Python is around 0.5 microseconds per operation (best case on Intel, worst case on Apple ARM). So that's maybe a 5% net loss if things were running synchronously. But they're not! When you call GPU code, you actually do it asynchronously, so the language latency can be completely hidden.
So really, all you want in an ML language is the ability to 1. change the type of the underlying data on the fly (Python is really good at this) and 2. rewrite the operations being dispatched to on the fly (Python is also really good at this).
For smaller models (i.e. things that run in sub-microsecond world), Python is not the right choice for training or deploying.
Earlier quoted context omitted.
> Since decoder-only transformer memory requirements scale with the square of sequence lengths, things would probably slow down significantly for very long sequences, which would be required for a back-and-forth conversation. You can use tricks to keep the sequence length down even if the conversation goes on for a long time. For example, you can use the model to summarize the first n-1 lines of the conversation and…
This is very interesting. Could you please elaborate and maybe share links to articles if you know of any?
This is cool! But I wonder if it's economical using cloud hardware. The author claims 1.12 tokens/second on the 175B parameter model (arguably comparable to GPT-3 Davinci). That's about 100k tokens a day on the GCP machine the author used. Someone double check my numbers here, but given the Davinci base cost of $0.02 per 1k tokens and GCP cost for the hardware listed "NVIIDA T4 (16GB) instance on GCP with 208GB of DR…
Thanks for running the cloud numbers on this. I ran some DIY numbers and they indicate less than a week to break even with the cloud, including all hardware and electricity costs. The cloud seems stupid expensive compared to running your own hardware for this kind of task.
This is cool! But I wonder if it's economical using cloud hardware. The author claims 1.12 tokens/second on the 175B parameter model (arguably comparable to GPT-3 Davinci). That's about 100k tokens a day on the GCP machine the author used. Someone double check my numbers here, but given the Davinci base cost of $0.02 per 1k tokens and GCP cost for the hardware listed "NVIIDA T4 (16GB) instance on GCP with 208GB of DR…
This is most likely aimed at people running models locally. And a homelab with 3090s/4090s is one or two orders of magnitude cheaper than GCP, if you use them continuously.