Exponentially faster language modelling • Hacker News Board

Re: Exponentially faster language modelling

#21

Post by nolist_policy » Wed, Nov 22, 2023, 2:14 PM UTC

iNic wrote: post #10

ndr wrote: post #4

Abstract: > Language models only really need to use an exponential fraction of their neurons for individual inferences. As proof, we present UltraFastBERT, a BERT variant that uses 0.3% of its neurons during inference while performing on par with similar BERT models. UltraFastBERT selectively engages just 12 out of 4095 neurons for each layer inference. This is achieved by replacing feedforward networks with fast fee…

Do I understand correctly that the difficulty of making this useful is writing code to run this idea on GPUs?

As far as I understood it: Forget GPUs, this thing is plenty fast on CPUs.

In general, GPUs are bad at branching. The fastest way to implement it on GPUs is probably to let it calculate both sides of the branch and then only use the result of the one that was taken. Which won't be faster than a normal NN.

Re: Exponentially faster language modelling

#22

Post by PaulHoule » Wed, Nov 22, 2023, 2:15 PM UTC

iNic wrote: post #10

ndr wrote: post #4

Abstract: > Language models only really need to use an exponential fraction of their neurons for individual inferences. As proof, we present UltraFastBERT, a BERT variant that uses 0.3% of its neurons during inference while performing on par with similar BERT models. UltraFastBERT selectively engages just 12 out of 4095 neurons for each layer inference. This is achieved by replacing feedforward networks with fast fee…

Do I understand correctly that the difficulty of making this useful is writing code to run this idea on GPUs?

They are getting a 78x speedup w/o hardware support which is pretty good: they think they can speed it up another 4x if they had the right hardware support. So it looks useful now with possibility to get better.

So long as I've been involved with neural networks for text analysis it's seemed to me that we really should be using sparse activations because any particular document only involves a limited set of concepts.

For instance a search engine for patents might be looking at a patent for adhesive tape which activates a certain set of concepts but is not going to activate concepts involved with bicycle derailleurs or public key cryptography: a sparse representation reflects this and dense representations don't.

Re: Exponentially faster language modelling

#23

Post by caerwy » Wed, Nov 22, 2023, 2:29 PM UTC

vouaobrasil wrote: post #20

This is rather scary. I feel we are witnessing the evolution of language models and artificial intelligence, which seems intellectually laudable until you realize that the underlying evolutionary framework for this evolution is the global capitalistic system whose only criteria for selection in short-term monetary gain. We are creating a monster.

[flagged]

Re: Exponentially faster language modelling

#24

Post by WithinReason » Wed, Nov 22, 2023, 2:41 PM UTC

WithinReason wrote: post #14

Link to previous paper: https://arxiv.org/abs/2308.14711 An attempt at a summary: They use a sigmoid function to make differentiable "soft" branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost. They gradually harden the branches so they become hard branches at the end of training. A br…

Also, this didn't come from OpenAI or DeepMind, or even industry. What are those guys even doing? :)

Re: Exponentially faster language modelling

#25

Post by [deleted] » Wed, Nov 22, 2023, 2:43 PM UTC

WithinReason wrote: post #13

yvdriess wrote: post #11

Earlier quoted context omitted.

Both. Cheaper CPU-based inference, GPUs are not as competitive for sparse linear algebra. This could lead to much larger models, as you only touch a small portion of the matrix during inference. However, the training here is still dense-LA on a GPU, so you still blow up the compute cost when increasing model size.

Note this doesn't speed up training

[deleted]

Re: Exponentially faster language modelling

#26

Post by hendler » Wed, Nov 22, 2023, 2:55 PM UTC

vouaobrasil wrote: post #20

This is rather scary. I feel we are witnessing the evolution of language models and artificial intelligence, which seems intellectually laudable until you realize that the underlying evolutionary framework for this evolution is the global capitalistic system whose only criteria for selection in short-term monetary gain. We are creating a monster.

Rather than looking to capitalism which has provided tremendous benefits to society as well as unintended consequences you may want to update your thinking to focus on the incentives alignment problem in general.

This TED talk articulates it well: https://youtu.be/WX_vN1QYgmE

What is after capitalism?

Re: Exponentially faster language modelling

#27

Post by ashirviskas » Wed, Nov 22, 2023, 2:59 PM UTC

algo_trader wrote: post #8

sdrg822 wrote: post #5

Cool. Important note: """ One may ask whether the conditionality introduced by the use of CMM does not make FFFs incompatible with the processes and hardware already in place for dense matrix multiplication and deep learning more broadly. In short, the answer is “No, it does not, save for some increased caching complexity." """ It's hard to beat the hardware lottery!

Infact, as stated in the paper, this is bad news > We therefore leave the attention layers untouched Meaning, presumably, that the GPU memory remains the bottleneck Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !!

>Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !!

I'm really interested, can you share where you got these numbers?

Re: Exponentially faster language modelling

#28

Post by mmaunder » Wed, Nov 22, 2023, 3:01 PM UTC

WithinReason wrote: post #24

WithinReason wrote: post #14

Link to previous paper: https://arxiv.org/abs/2308.14711 An attempt at a summary: They use a sigmoid function to make differentiable "soft" branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost. They gradually harden the branches so they become hard branches at the end of training. A br…

Also, this didn't come from OpenAI or DeepMind, or even industry. What are those guys even doing? :)

Many labs doing foundational work like this and making progress don’t have the anything near the budget or compute to implement at scale. In other words they don’t have a Sam and his backers or a Zuck and his budget.

Re: Exponentially faster language modelling

#29

Post by Micoloth » Wed, Nov 22, 2023, 3:05 PM UTC

WithinReason wrote: post #24

WithinReason wrote: post #14

Link to previous paper: https://arxiv.org/abs/2308.14711 An attempt at a summary: They use a sigmoid function to make differentiable "soft" branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost. They gradually harden the branches so they become hard branches at the end of training. A br…

Also, this didn't come from OpenAI or DeepMind, or even industry. What are those guys even doing? :)

They sure as hell have no incentives to make Neural Network faster and more accessible, for starters..

(Considering they right now make more money and have more control, the less accessible and the more computation-hungry AI models are)

To be fair, this approach (claims to) only speed up inference, not training, so all the GPUs are needed anyway.

Re: Exponentially faster language modelling

#30

Post by WithinReason » Wed, Nov 22, 2023, 3:09 PM UTC

Micoloth wrote: post #29

WithinReason wrote: post #24

Earlier quoted context omitted.

Also, this didn't come from OpenAI or DeepMind, or even industry. What are those guys even doing? :)

They sure as hell have no incentives to make Neural Network faster and more accessible, for starters.. (Considering they right now make more money and have more control, the less accessible and the more computation-hungry AI models are) To be fair, this approach (claims to) only speed up inference, not training, so all the GPUs are needed anyway.

They certainly have an incentive to keep these kinds of improvements in-house and not publish them, since they are commercial entities and this represents a competitive advantage.