Earlier quoted context omitted.
There's another paper replacing attention with FF networks so just combine the two and you've got something.
Link? Sounds like a good read! :)
Not op but might be this: https://arxiv.org/pdf/2311.10642.pdf
Exponentially faster language modelling
101–110 of 150 posts
Re: Exponentially faster language modelling
#102hugging face model https://huggingface.co/pbelcak/UltraFastBERT-1x11-long
Is it possible to use this with something like Llama 2?
Re: Exponentially faster language modelling
#103I find running 7B models on my 6 year old small form factor HP EliteDesk to be fast enough for casual everyday use. If this speedup can be applied generally to commonly used models, I can serve a local ChatGPT experience for both friends and family from my tiny homelab in my basement. mind blown
I find 7B models to be too stupid. They often respond with nonsense or fail to follow instructions.
Even 65B models approach a level of being almost usable, but still fall short in my personal experience.
Re: Exponentially faster language modelling
#104Link to previous paper: https://arxiv.org/abs/2308.14711 An attempt at a summary: They use a sigmoid function to make differentiable "soft" branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost. They gradually harden the branches so they become hard branches at the end of training. A br…
Also, this didn't come from OpenAI or DeepMind, or even industry. What are those guys even doing? :)
This from DeepMind:
DiLoCo: Distributed Low-Communication Training of Language Models - https://arxiv.org/pdf/2311.08105.pdf
From the first author on Twitter: "It could quite a big deal for people who don't have access to a colocated cluster of GPUs:
e.g. with DiLoCo you could train your model, with data-parallelism, across all GPU providers, looking in real-time for the cheapest price, even if pre-emptable, even across continents"
Re: Exponentially faster language modelling
#105If anyone is on the ball enough to turn this into a colab or notebook that would be appreciated! Would love to see the code
Re: Exponentially faster language modelling
#106Earlier quoted context omitted.
I find 7B models to be too stupid. They often respond with nonsense or fail to follow instructions.
Even 65B models approach a level of being almost usable, but still fall short in my personal experience.
This is why I'm not understanding the excitement around open source models, they pale in comparison to GPT-4 quality, so I have no use for them until we have something comparable.
Re: Exponentially faster language modelling
#107Earlier quoted context omitted.
They sure as hell have no incentives to make Neural Network faster and more accessible, for starters.. (Considering they right now make more money and have more control, the less accessible and the more computation-hungry AI models are) To be fair, this approach (claims to) only speed up inference, not training, so all the GPUs are needed anyway.
I wouldn't be so quick to conspiracy. I'm the author of a work and a famous blog post that trains a particular common architecture much faster (don't want to dox myself too much) and with far fewer parameters, but it has been rejected several times and is now arxiv only. Our most common complaint was "who would use this? Why not just take a large model and tune it?" That question alone held us back a year (had over a…
Is there a place where you guys discuss... things? I'm layman interested in this topic akin to pop-physics/maths, but have no chance to just read papers and "get it". On the other hand, immediately available resources focus more on how-to part of it rather than on what's up overall. Also, do you have something like 3b1b/pbs/nph for it? Content that you can watch and say "well, yep, good job".
Re: Exponentially faster language modelling
#108Mix this with yesterday's matmul approximation (maddness) in HW for a casual... three orders of magnitude speed increase?
They are mostly incompatible.
Re: Exponentially faster language modelling
#109Link to previous paper: https://arxiv.org/abs/2308.14711 An attempt at a summary: They use a sigmoid function to make differentiable "soft" branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost. They gradually harden the branches so they become hard branches at the end of training. A br…
It's mentioned briefly in the paper(1), but I'm more interested in the interpretability implications of this approach. In some respects, this marries the interpretability/editability of a small decision tree with the expressive power of a large neural network. Usually you see those two on extreme opposite ends of a tradeoff spectrum - but this approach, if it scales, might shift the pareto frontier.
(1): As a byproduct, the learned regions can also be used as a partition of the input space for interpretability, surgical model editing, catastrophic forgetting mitigation, reduction of replay data budget, etc..
Re: Exponentially faster language modelling
#110I find running 7B models on my 6 year old small form factor HP EliteDesk to be fast enough for casual everyday use. If this speedup can be applied generally to commonly used models, I can serve a local ChatGPT experience for both friends and family from my tiny homelab in my basement. mind blown
I find 7B models to be too stupid. They often respond with nonsense or fail to follow instructions.
Even like OpenChat-3.5? (Probably the best 7B model out there) Demo: https://openchat.team/
HuggingFace: https://huggingface.co/openchat/openchat_3.5
On the LLM arena (blinded comparisons), it's the third best non-proprietary model: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...