Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exponentially faster language modelling

arxiv.org

Post reply on HN
Print view View original

61–70 of 150 posts

Re: Exponentially faster language modelling

#61

Post by vouaobrasil »

hendler wrote: post #26
vouaobrasil wrote: post #20

This is rather scary. I feel we are witnessing the evolution of language models and artificial intelligence, which seems intellectually laudable until you realize that the underlying evolutionary framework for this evolution is the global capitalistic system whose only criteria for selection in short-term monetary gain. We are creating a monster.

Rather than looking to capitalism which has provided tremendous benefits to society as well as unintended consequences you may want to update your thinking to focus on the incentives alignment problem in general. This TED talk articulates it well: https://youtu.be/WX_vN1QYgmE What is after capitalism?

I absolutely disagree that reformism as in the video via incentives will be enough.

Re: Exponentially faster language modelling

#62

Post by vouaobrasil »

dicroce wrote: post #32
vouaobrasil wrote: post #20

This is rather scary. I feel we are witnessing the evolution of language models and artificial intelligence, which seems intellectually laudable until you realize that the underlying evolutionary framework for this evolution is the global capitalistic system whose only criteria for selection in short-term monetary gain. We are creating a monster.

I think it's good to be concerned and cautious but I also think you are being a bit extreme here.

I absolutely disagree - I believe everyone else is blind, the same way we are blind that our current lifestyles are an exercise in extreme violence on the nonhuman world.

Re: Exponentially faster language modelling

#65

Post by hobofan »

baq wrote: post #51

Mix this with yesterday's matmul approximation (maddness) in HW for a casual... three orders of magnitude speed increase?

I'm not 100% sure, but those seem mostly mutually exclusive (or redundant), with the decision tree in maddness taking on a similar function as the binary tree in FFF that decides which neurons to activate.

Re: Exponentially faster language modelling

#66

Post by lopuhin »

millisecond wrote: post #7

Could this be applied to other models like Llama2 or Mistral?

There are two issues here -- for one, in big transformers, more compute is in the attention layers, while this work improves only feed-forward layers, which are more important for smaller models and smaller sequence lengths. Second, in many typical scenarios LLM inference is memory bandwidth bound, I'm not sure if it's possible to utilize their approach to reduce required memory bandwidth.

Re: Exponentially faster language modelling

#67

Post by hobofan »

lawlessone wrote: post #38
nolist_policy wrote: post #21

Earlier quoted context omitted.

As far as I understood it: Forget GPUs, this thing is plenty fast on CPUs. In general, GPUs are bad at branching. The fastest way to implement it on GPUs is probably to let it calculate both sides of the branch and then only use the result of the one that was taken. Which won't be faster than a normal NN.

> Forget GPUs, this thing is plenty fast on CPUs. Does this mean everyone could be running the 100+b models from ram? This opens up a lot , some models could be run very fast on small machines with this. Bundling a small model inside a game to act as part of the mind for ingame NPC's (obviously with some tuning) becomes practical with this.

The bottleneck for "easy integration" into games and applications right now is as much the RAM usage as is the slowness. This would probably bring the speed to an acceptable level but you would still have to hold the whole model in RAM.

That would make it a lot more feasible to run models in the cloud (triple digit RAM is a lot more abundant than VRAM), but wouldn't do that much for consumer hardware.

Re: Exponentially faster language modelling

#68

Post by nolist_policy »

hobofan wrote: post #67
lawlessone wrote: post #38

Earlier quoted context omitted.

> Forget GPUs, this thing is plenty fast on CPUs. Does this mean everyone could be running the 100+b models from ram? This opens up a lot , some models could be run very fast on small machines with this. Bundling a small model inside a game to act as part of the mind for ingame NPC's (obviously with some tuning) becomes practical with this.

The bottleneck for "easy integration" into games and applications right now is as much the RAM usage as is the slowness. This would probably bring the speed to an acceptable level but you would still have to hold the whole model in RAM. That would make it a lot more feasible to run models in the cloud (triple digit RAM is a lot more abundant than VRAM), but wouldn't do that much for consumer hardware.

I wonder if the model takes similar branches while in the same context? Then you can fault in parts of the model from disk as needed.

Re: Exponentially faster language modelling

#69

Post by entropicdrifter »

marcinzm wrote: post #18
algo_trader wrote: post #8

Earlier quoted context omitted.

Infact, as stated in the paper, this is bad news > We therefore leave the attention layers untouched Meaning, presumably, that the GPU memory remains the bottleneck Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !!

Bottleneck for larger models however this would presumably allow for cheaper models at scale or on compute constrained devices (like phones).

And potentially for distributing a model across several devices at inference time. You could devote a cluster of smaller/weaker machines to inference.

Re: Exponentially faster language modelling

#70

Post by joelthelion »

lopuhin wrote: post #66
millisecond wrote: post #7

Could this be applied to other models like Llama2 or Mistral?

There are two issues here -- for one, in big transformers, more compute is in the attention layers, while this work improves only feed-forward layers, which are more important for smaller models and smaller sequence lengths. Second, in many typical scenarios LLM inference is memory bandwidth bound, I'm not sure if it's possible to utilize their approach to reduce required memory bandwidth.

Doesn't reducing the number of neurons drastically reduce memory requirements?
Post reply on HN