141–150 of 334 posts
Earlier quoted context omitted.
afaik, gpt-4 is mostly rumours so far, same thing for the 1.5T number. gpt-4 is suerly coming.
Maybe it will be called GPT-XP by then, with Microsoft owning half of it.
Earlier quoted context omitted.
Some repos have code that 'phones home' when run. For example, checking for updates or security vulnerabilities. By checking the usage statistics on that server, you can get an idea how many users there are, and typically it's far higher than the number of stars.
That just tells us that more people use the code than star the repo. I don’t think that’d be a surprise to anyone. The claim was that more people clone and copy code from the repo than the ones who star it, which is a different matter from the number of users.
This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.
So if you buy two of these cards it will take 12-13 days instead of 38 hours but only require a $2500 PC.
James Betker, who created tortoise TTS, built his own $15k machine with 8x RTX 3090 and trained the models with it. He now works for OpenAI…
Earlier quoted context omitted.
You can't know that. Currently, 8 billion humans generate a few scientific breakthroughs per year. You'd have to run several billion ChatGPTs for a year with zero breakthroughs to have any confidence in such a claim.
At this point there is no framework that suggests GPT understands the underlying data. It can’t assign meaning as a human would. It can’t consume hundreds of math textbooks and learn the principles of math and then apply them more broadly to science textbooks and research papers. It can’t even reliably add two numbers. Yes, brute forcing with hard AI can produce many thoughts. But the AI wouldn’t know they are correc…
Actually there are many indications that GPT understands the data, because its output mostly makes sense. The reason it can't assign meaning the way a human would is because a human can correlate words with other sensory data that GPT doesn't have access to. That's where GPT creates nonsense.
Think carefully about what "understanding" means in a mechanistic sense. It's a form of compression, and a few billion parameters encoding the contents of a large part of the internet seems like pretty good compression to me.
Earlier quoted context omitted.
As models get bigger less and less neurons get activated by any given input. If you can somehow predict which neurons get activated you can skip the vast majority of the computational load. I have read a paper where they argued that only 0.5% of the neurons are actually active in a 200 million parameter model so you can get a 200x improvement just from that. What this tells you is that there is very little money in o…
Do you have a link to that paper by any chance? By "neurons" did they mean weights or activations?
https://ieeexplore.ieee.org/document/9635657
It is somewhere from 8x to 25x faster than doing dense machine learning. The speedup was higher on the original CPU implementation and the GPU paper mentions that if there isn't enough shared memory on the GPU it will have to switch to an algorithm that has more overhead.
By neurons I actually meant "nodes"
My comment is effectively a summary of this article: https://www.kdnuggets.com/2020/03/deep-learning-breakthrough...
Edit: There is a paper for sparse spiking gradient descent promising a 150x improvement. I am not sure how practical this is because spiking neural network hardware heavily limits your model size but here it is:
This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.
How are universities and colleges dealing with this kind of demand for computing power? It must be hard to be able to do some courses now.
Are there any possible technologal or scientific leaps on the horizon that would reduce training time by an order of magnitude or more? GPT-3 took 355 years to train with incredibly expensive hardware, which means small players have no chance to push the state of the art
As models get bigger less and less neurons get activated by any given input. If you can somehow predict which neurons get activated you can skip the vast majority of the computational load. I have read a paper where they argued that only 0.5% of the neurons are actually active in a 200 million parameter model so you can get a 200x improvement just from that. What this tells you is that there is very little money in o…