331–334 of 334 posts
Earlier quoted context omitted.
Intelligent but not conscious would still be a few steps ahead of GPT. We can take a concept and refactor it symbolically. GPT can't do that. All it does is find symbols that are semantically close to other symbols.
I’m not sure that those two processes are as distinct as you believe them to be.
That's circular reasoning.
Are there any possible technologal or scientific leaps on the horizon that would reduce training time by an order of magnitude or more? GPT-3 took 355 years to train with incredibly expensive hardware, which means small players have no chance to push the state of the art
As models get bigger less and less neurons get activated by any given input. If you can somehow predict which neurons get activated you can skip the vast majority of the computational load. I have read a paper where they argued that only 0.5% of the neurons are actually active in a 200 million parameter model so you can get a 200x improvement just from that. What this tells you is that there is very little money in o…
This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.
If you can fit the training into 24GB, a used RTX 3090 for $700-$800 seems like a good deal at the moment. They are about 45-65% as fast as the A100 according to https://bizon-tech.com/gpu-benchmarks/NVIDIA-RTX-3090-vs-NVI... So if you buy two of these cards it will take 12-13 days instead of 38 hours but only require a $2500 PC. James Betker, who created tortoise TTS, built his own $15k machine with 8x RTX 3090 and…
https://timdettmers.com/2023/01/16/which-gpu-for-deep-learni...
TL;DR: You probably don't need that expensive Threadripper because 2x PCIe 4.0 x16 will not be very beneficial. Go cheap, go 2x PCIe 4.0 x8.