71–80 of 89 posts
I got into deep learning around when ReLU and dropout was hot and on my consumer 1080 I was able to change one or two lines of code and test the improvements in a few hours, whereas now, I guess I'll need to wait a few weeks for mistral et al to try it out
Maybe the only downside to how fast LLMs are moving is papers come out faster than anyone (not at Google) can train and test the improvements. I got into deep learning around when ReLU and dropout was hot and on my consumer 1080 I was able to change one or two lines of code and test the improvements in a few hours, whereas now, I guess I'll need to wait a few weeks for mistral et al to try it out
I'm focusing in quantization approaches and testing on my obsolete last gen GPUs.
Maybe the only downside to how fast LLMs are moving is papers come out faster than anyone (not at Google) can train and test the improvements. I got into deep learning around when ReLU and dropout was hot and on my consumer 1080 I was able to change one or two lines of code and test the improvements in a few hours, whereas now, I guess I'll need to wait a few weeks for mistral et al to try it out
Welcome to the GPU poor! I'm focusing in quantization approaches and testing on my obsolete last gen GPUs.
Most important paper of 2024. The idea that we want models not to have to use the same amount of compute for every token has been around for a while. This is the first compelling mechanism I've seen for doing it. > Equipped with these new methods, we can sample autoregressively by choosing to route tokens to or around a block based on the router’s output, which does not depend on any information from future tokens. W…
Most important? The idea that not every token needs the full context window should be an obvious optimization.
hu-po does in-depth live-stream reviews of AI papers. highly recommended, here is his take on the mixture-of-depths paper discussed. https://www.youtube.com/watch?v=Teru_qIdB8Y
Earlier quoted context omitted.
Speculative decoding does this to an extent - using a smaller model to generate its own predictions and putting them in the batch of the bigger model until they diverge https://huggingface.co/blog/whisper-speculative-decoding
It doesn’t. It simply trades compute efficiency by transposing matrix multiplications into “the future.” It doesn’t actually save FLOPs (uses more) and doesn’t work at large batch size
Does anyone even care? Really, who cares? The truth is nobody cares. Saving FLOPs does nothing if you have to load the entire model anyway. Going from two flops per parameter to 0.5 or whatever might sound cool on paper but you're loading those parameters anyway and gained nothing.
I wrote up a bit about it here, from what I could piece together: https://lifeinthesingularity.com/p/googles-breakthroughs-in-...
Nice writing. Reminds me of New Scientist style. (I like NS so that is a compliment). I think the “explain as you go along but be brief style”. Which is nice for getting a feel for the space.
i try to operate at zero-basis and quickly scaffold to a simplified model so ANYONE can grab the "why this makes nerds say wow" factor haha
Earlier quoted context omitted.
i know you're jesting but RNNs are recursive along the sequence length where I am describing recursion along the depth.
Recursive NNs are not the same as Recurrent NNs: https://en.wikipedia.org/wiki/Recursive_neural_network Well ish. The article above explains that Recursive-NNs are hierarchical whereas RNNs are linear. I guess the distinction is a little on the fine side. Anyway carry on. Pedantic moment over.
Earlier quoted context omitted.
Recursive NNs are not the same as Recurrent NNs: https://en.wikipedia.org/wiki/Recursive_neural_network Well ish. The article above explains that Recursive-NNs are hierarchical whereas RNNs are linear. I guess the distinction is a little on the fine side. Anyway carry on. Pedantic moment over.
The recursive neural networks described there are a failed academic project from more than a decade ago, predating modern deep learning. Basically everyone using the phrase recursive nn nowadays is probably just mispeaking for RNN. RNNs also are not linear