Nvidia predicts AI models 1,000,000X more powerful than ChatGPT within 10 years

Post by [deleted] » Sun, Feb 26, 2023, 2:27 PM UTC

zamadatix wrote: post #34

brhsagain wrote: post #29

I have a probably really stupid question but I'm going to ask it because I really don't know the answer. What's stopping someone from starting a distributed effort to train an open source ChatGPT model, like Folding@home but for AI? Is it a technical limitation with how models are trained, difficulty of coordination, something else?

The models are extremely large to ship around, sync, and load on consumer hardware over the internet. Maybe there are better ways to build and train a model for a distributed use case though.

Post by imtringued » Sun, Feb 26, 2023, 6:37 PM UTC

ninethirty wrote: post #45

epups wrote: post #14

There is not enough written text to scale a model like ChatGPT at even 10x its current parameters, so improvements will have to come from optimizations and/or longer training (at diminishing returns).

GPT-4 Will Have 100 Trillion Parameters — 500x the Size of GPT-3

Post by brokenmachine » Tue, Feb 28, 2023, 5:24 AM UTC

d4rkp4ttern wrote: post #56

Ok I’ll be the one to ask — 1,000,000x means what, exactly?

Post by zamadatix » Thu, Mar 02, 2023, 2:21 AM UTC

zirgs wrote: post #64

KeplerBoy wrote: post #53

Earlier quoted context omitted.

which would make traditional distributed training prohibitively slow. You would have to sync the weights with residential internet speeds (10s of MB/s) instead of PCI-e Speeds (10s of GB/s).

If you want to add new stuff to Stable Diffusion - you don't have to retrain the main model from scratch. You can train it only on a few hundred images at a time and then add the resulting model as an extension or merge it into the main model. People train their models on a single celebrity or a single artist that way. Other types of AI could be trained in a similar way.