71–80 of 150 posts
Earlier quoted context omitted.
Infact, as stated in the paper, this is bad news > We therefore leave the attention layers untouched Meaning, presumably, that the GPU memory remains the bottleneck Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !!
There's another paper replacing attention with FF networks so just combine the two and you've got something.
Earlier quoted context omitted.
From the previous paper you cited >Pushing FFFs to the limit, we show that they can use as little as 1% of layer neurons for inference in vision transformers while preserving 94.2% of predictive performance. This feels like that often misinterpreted Einstein meme/qoute about humans only using a fraction of their brain power. Is this only for inference though? could it boost training?
That's an interesting question. It actually provides a nice way to parallelized training: Pretrain e.g. the first 3 branch levels, which effectively fragments the model into 8 separate parts, which you can continue training across 8 independent servers/nodes with no further communication between the nodes. A central server would run the 1st 3 levels and mark parts of the training set that each node has to train on. M…
Earlier quoted context omitted.
The bottleneck for "easy integration" into games and applications right now is as much the RAM usage as is the slowness. This would probably bring the speed to an acceptable level but you would still have to hold the whole model in RAM. That would make it a lot more feasible to run models in the cloud (triple digit RAM is a lot more abundant than VRAM), but wouldn't do that much for consumer hardware.
I wonder if the model takes similar branches while in the same context? Then you can fault in parts of the model from disk as needed.
Why is the context size and batch size represented as a single parameter?
Earlier quoted context omitted.
From the previous paper you cited >Pushing FFFs to the limit, we show that they can use as little as 1% of layer neurons for inference in vision transformers while preserving 94.2% of predictive performance. This feels like that often misinterpreted Einstein meme/qoute about humans only using a fraction of their brain power. Is this only for inference though? could it boost training?
That's an interesting question. It actually provides a nice way to parallelized training: Pretrain e.g. the first 3 branch levels, which effectively fragments the model into 8 separate parts, which you can continue training across 8 independent servers/nodes with no further communication between the nodes. A central server would run the 1st 3 levels and mark parts of the training set that each node has to train on. M…
Does this also mean that it would be possible to train on parallel GPU-poor setup instead of needing lots of GPU memory / bandwidth on one computer?
Does anyone understand why they are using B x H instead of B x S x H? Why is the context size and batch size represented as a single parameter?
Earlier quoted context omitted.
They certainly have an incentive to keep these kinds of improvements in-house and not publish them, since they are commercial entities and this represents a competitive advantage.
I think Nvidia might have an incentive for this not to exist. edit: but you are right for the AI companies not open sourcing their models it's an advantage to have it when others don't
Earlier quoted context omitted.
From the previous paper you cited >Pushing FFFs to the limit, we show that they can use as little as 1% of layer neurons for inference in vision transformers while preserving 94.2% of predictive performance. This feels like that often misinterpreted Einstein meme/qoute about humans only using a fraction of their brain power. Is this only for inference though? could it boost training?
That's an interesting question. It actually provides a nice way to parallelized training: Pretrain e.g. the first 3 branch levels, which effectively fragments the model into 8 separate parts, which you can continue training across 8 independent servers/nodes with no further communication between the nodes. A central server would run the 1st 3 levels and mark parts of the training set that each node has to train on. M…
Earlier quoted context omitted.
Also, this didn't come from OpenAI or DeepMind, or even industry. What are those guys even doing? :)
…ETH Zurich is an illustrious research university that often cooperates with Deepmind and other hyped groups, they're right there at the frontier too, and have been for a very long time. They don't have massive training runs on their own but pound for pound I'd say they have better papers.
It's easy to get hyped up on the big tech labs because they have the most compute, but the best papers come from smaller labs and unfortunately more lately face larger challenges in getting published. It's the smaller works that create the foundations that end up in these giant models. ML is in a really weird space right now.
Earlier quoted context omitted.
From the previous paper you cited >Pushing FFFs to the limit, we show that they can use as little as 1% of layer neurons for inference in vision transformers while preserving 94.2% of predictive performance. This feels like that often misinterpreted Einstein meme/qoute about humans only using a fraction of their brain power. Is this only for inference though? could it boost training?
That's an interesting question. It actually provides a nice way to parallelized training: Pretrain e.g. the first 3 branch levels, which effectively fragments the model into 8 separate parts, which you can continue training across 8 independent servers/nodes with no further communication between the nodes. A central server would run the 1st 3 levels and mark parts of the training set that each node has to train on. M…
Well, I'm assuming they'd use whatever they're given, so maybe the question should be "how much less time would training take on a 1.5 exaflops computer?"