Mix this with yesterday's matmul approximation (maddness) in HW for a casual... three orders of magnitude speed increase?
Exponentially faster language modelling
51–60 of 150 posts
Re: Exponentially faster language modelling
#52Cool. Important note: """ One may ask whether the conditionality introduced by the use of CMM does not make FFFs incompatible with the processes and hardware already in place for dense matrix multiplication and deep learning more broadly. In short, the answer is “No, it does not, save for some increased caching complexity." """ It's hard to beat the hardware lottery!
Infact, as stated in the paper, this is bad news > We therefore leave the attention layers untouched Meaning, presumably, that the GPU memory remains the bottleneck Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !!
There's another paper replacing attention with FF networks so just combine the two and you've got something.
Re: Exponentially faster language modelling
#53Mix this with yesterday's matmul approximation (maddness) in HW for a casual... three orders of magnitude speed increase?
Can you link the post for the matmul approximation?
Re: Exponentially faster language modelling
#54Earlier quoted context omitted.
I think Nvidia might have an incentive for this not to exist. edit: but you are right for the AI companies not open sourcing their models it's an advantage to have it when others don't
I'm actually not sure about Nvidia, due to https://en.wikipedia.org/wiki/Jevons_paradox
But if things get too efficient for individual users, you won't need an Nvidia GPU anymore. People will use cheaper hardware instead. I'm looking forward to running good models at decent speed on a low-end CPU or whatever crappy GPU is in my phone.
Re: Exponentially faster language modelling
#55Earlier quoted context omitted.
From the previous paper you cited >Pushing FFFs to the limit, we show that they can use as little as 1% of layer neurons for inference in vision transformers while preserving 94.2% of predictive performance. This feels like that often misinterpreted Einstein meme/qoute about humans only using a fraction of their brain power. Is this only for inference though? could it boost training?
That's an interesting question. It actually provides a nice way to parallelized training: Pretrain e.g. the first 3 branch levels, which effectively fragments the model into 8 separate parts, which you can continue training across 8 independent servers/nodes with no further communication between the nodes. A central server would run the 1st 3 levels and mark parts of the training set that each node has to train on. M…
This!
If this becomes true then it’s a game changer. I hope you are correct.
Re: Exponentially faster language modelling
#56According to scientists, we only use 0.3% of our neural networks. Imagine if we could use 100%.
[dead]
Re: Exponentially faster language modelling
#57Earlier quoted context omitted.
Both. Cheaper CPU-based inference, GPUs are not as competitive for sparse linear algebra. This could lead to much larger models, as you only touch a small portion of the matrix during inference. However, the training here is still dense-LA on a GPU, so you still blow up the compute cost when increasing model size.
Has anyone used SIMD instructions to try and speed up cpu inference?
A lot of CPU inference libraries (llama.cpp included) use as much SIMD as possible, sometimes by hand-writing loops. The one I hack on, llama.rs, uses portable_simd but specializes to your CPU at compile time.
My experience has been that most CPU inference is actually not compute limited, but memory bandwidth limited, since most weights are used for a few operations per token (how quickly can you load and unload the entire 70 GB of weights into your registers?). It's not quite that bad but I found most vectorization changes didn't meaningfully change performance.
Re: Exponentially faster language modelling
#58Could this be applied to other models like Llama2 or Mistral?
It certainly could, and I wouldn't be surprised if the authors want to try it out on those. You do have issues of past improvements often not quite enhancing more powerful models nearly as much. I'd expect this to possibly not work as well, something like the bigger models ending up with more polysemantic neurons because they're given more ''incentive'' (training time, neuron count, dataset size which they're encouraged to be able to reconstruct) to extract as much possible. This might make so the method performs worse due to this intermingling.
(See the transformer circuits website for that)
(Though I expect there's ways to recover a good chunk of extra lost throughput/accuracy, maybe by doing extra steps to directly steer the training towards breaking apart polysemantic neurons)
Re: Exponentially faster language modelling
#59Link to previous paper: https://arxiv.org/abs/2308.14711 An attempt at a summary: They use a sigmoid function to make differentiable "soft" branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost. They gradually harden the branches so they become hard branches at the end of training. A br…
Also, this didn't come from OpenAI or DeepMind, or even industry. What are those guys even doing? :)
…ETH Zurich is an illustrious research university that often cooperates with Deepmind and other hyped groups, they're right there at the frontier too, and have been for a very long time. They don't have massive training runs on their own but pound for pound I'd say they have better papers.
Re: Exponentially faster language modelling
#60How would this scale for a use case like writing code? I could imagine that some inputs would require a large number of neurons. Would this architecture be able to do that if it were scaled up?
I'm also curious if this model architecture would achieve the grokking of more complex concepts at scale.