41–50 of 150 posts
Abstract: > Language models only really need to use an exponential fraction of their neurons for individual inferences. As proof, we present UltraFastBERT, a BERT variant that uses 0.3% of its neurons during inference while performing on par with similar BERT models. UltraFastBERT selectively engages just 12 out of 4095 neurons for each layer inference. This is achieved by replacing feedforward networks with fast fee…
According to scientists, we only use 0.3% of our neural networks. Imagine if we could use 100%.
Link to previous paper: https://arxiv.org/abs/2308.14711 An attempt at a summary: They use a sigmoid function to make differentiable "soft" branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost. They gradually harden the branches so they become hard branches at the end of training. A br…
This feels like that often misinterpreted Einstein meme/qoute about humans only using a fraction of their brain power.
Is this only for inference though? could it boost training?
According to scientists, we only use 0.3% of our neural networks. Imagine if we could use 100%.
I know HN can sometimes be the place where humor goes to die, but I found this comment hilarious.
Earlier quoted context omitted.
They sure as hell have no incentives to make Neural Network faster and more accessible, for starters.. (Considering they right now make more money and have more control, the less accessible and the more computation-hungry AI models are) To be fair, this approach (claims to) only speed up inference, not training, so all the GPUs are needed anyway.
They certainly have an incentive to keep these kinds of improvements in-house and not publish them, since they are commercial entities and this represents a competitive advantage.
edit: but you are right for the AI companies not open sourcing their models it's an advantage to have it when others don't
hugging face model https://huggingface.co/pbelcak/UltraFastBERT-1x11-long
> This model is provided only as sanity check for research purposes, it is untested and unfit for deployment. I guess this means it isn't pretrained yet? Is it still just random weights?
It doesn't mean that there's anything technically wrong with the language model per se as a model of language, just that there has been no effort made to ensure it's fit to be deployed as-is for any given generative-AI use case, and the model authors would prefer you didn't do that.
Link to previous paper: https://arxiv.org/abs/2308.14711 An attempt at a summary: They use a sigmoid function to make differentiable "soft" branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost. They gradually harden the branches so they become hard branches at the end of training. A br…
Link to previous paper: https://arxiv.org/abs/2308.14711 An attempt at a summary: They use a sigmoid function to make differentiable "soft" branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost. They gradually harden the branches so they become hard branches at the end of training. A br…
From the previous paper you cited >Pushing FFFs to the limit, we show that they can use as little as 1% of layer neurons for inference in vision transformers while preserving 94.2% of predictive performance. This feels like that often misinterpreted Einstein meme/qoute about humans only using a fraction of their brain power. Is this only for inference though? could it boost training?
Hold on, you don't even need to freeze the branches completely: each node could train 1 branch on the path to its leaf node and communicate a change in the branch node to a central server, so you can distribute training without having to pre-freeze the branches. Still would need some pre-training though, and the splits would change slowly, and the attention mechanism could complicate things.
Currently distributed neural network training SETI@HOME style looks like a complete pipe dream that nobody is taking seriously. But a smart branching mechanism like this could suddenly make it possible. Folding@home reached 1.5 exaflops, which made it the world's largest supercomputer. Imagine the models we could train that way, they would far surpass whatever OpenAI or Google could train and would be public.
Earlier quoted context omitted.
They certainly have an incentive to keep these kinds of improvements in-house and not publish them, since they are commercial entities and this represents a competitive advantage.
I think Nvidia might have an incentive for this not to exist. edit: but you are right for the AI companies not open sourcing their models it's an advantage to have it when others don't
Earlier quoted context omitted.
>Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !! I'm really interested, can you share where you got these numbers?
Axelera [1] or Halio [2] give you 100-200tflop for ~$200. 8-bit ops, inference only, low memory embedded, excluding the host, implied utilization from FPS specs is ~20% But the trend is there. There are also newer ADAS/AV units from China which claim 1000tflops and cant really cost more than $1000/$2000 per car. These are all tiled designed (see also dojo/tesla) heavily over-weighed on flops vs memory [1] https://www…