Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exponentially faster language modelling

arxiv.org

Post reply on HN
Print view View original

31–40 of 150 posts

Re: Exponentially faster language modelling

#31

Post by falcor84 »

nolist_policy wrote: post #21
iNic wrote: post #10

Earlier quoted context omitted.

Do I understand correctly that the difficulty of making this useful is writing code to run this idea on GPUs?

As far as I understood it: Forget GPUs, this thing is plenty fast on CPUs. In general, GPUs are bad at branching. The fastest way to implement it on GPUs is probably to let it calculate both sides of the branch and then only use the result of the one that was taken. Which won't be faster than a normal NN.

I wonder then, if each inference only uses a small part of the net, could you possibly perform multiple inferences in the same forward pass?

Re: Exponentially faster language modelling

#32

Post by dicroce »

vouaobrasil wrote: post #20

This is rather scary. I feel we are witnessing the evolution of language models and artificial intelligence, which seems intellectually laudable until you realize that the underlying evolutionary framework for this evolution is the global capitalistic system whose only criteria for selection in short-term monetary gain. We are creating a monster.

I think it's good to be concerned and cautious but I also think you are being a bit extreme here.

Re: Exponentially faster language modelling

#33

Post by swalsh »

yvdriess wrote: post #11
Klaster_1 wrote: post #6

What are the potential consequences? Does this open doors to faster edge inference or improved capabilities?

Both. Cheaper CPU-based inference, GPUs are not as competitive for sparse linear algebra. This could lead to much larger models, as you only touch a small portion of the matrix during inference. However, the training here is still dense-LA on a GPU, so you still blow up the compute cost when increasing model size.

Has anyone used SIMD instructions to try and speed up cpu inference?

Re: Exponentially faster language modelling

#34

Post by [deleted] »

hendler wrote: post #26
vouaobrasil wrote: post #20

This is rather scary. I feel we are witnessing the evolution of language models and artificial intelligence, which seems intellectually laudable until you realize that the underlying evolutionary framework for this evolution is the global capitalistic system whose only criteria for selection in short-term monetary gain. We are creating a monster.

Rather than looking to capitalism which has provided tremendous benefits to society as well as unintended consequences you may want to update your thinking to focus on the incentives alignment problem in general. This TED talk articulates it well: https://youtu.be/WX_vN1QYgmE What is after capitalism?

[deleted]

Re: Exponentially faster language modelling

#35

Post by algo_trader »

ashirviskas wrote: post #27
algo_trader wrote: post #8

Earlier quoted context omitted.

Infact, as stated in the paper, this is bad news > We therefore leave the attention layers untouched Meaning, presumably, that the GPU memory remains the bottleneck Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !!

>Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !! I'm really interested, can you share where you got these numbers?

Axelera [1] or Halio [2] give you 100-200tflop for ~$200.

8-bit ops, inference only, low memory embedded, excluding the host, implied utilization from FPS specs is ~20%

But the trend is there.

There are also newer ADAS/AV units from China which claim 1000tflops and cant really cost more than $1000/$2000 per car.

These are all tiled designed (see also dojo/tesla) heavily over-weighed on flops vs memory

[1] https://www.axelera.ai/

[2] https://hailo.ai/

Re: Exponentially faster language modelling

#36

Post by hobofan »

swalsh wrote: post #33
yvdriess wrote: post #11

Earlier quoted context omitted.

Both. Cheaper CPU-based inference, GPUs are not as competitive for sparse linear algebra. This could lead to much larger models, as you only touch a small portion of the matrix during inference. However, the training here is still dense-LA on a GPU, so you still blow up the compute cost when increasing model size.

Has anyone used SIMD instructions to try and speed up cpu inference?

Most inference builds on top of BLAS libraries, which in their implementation take advantage of SIMD.

Re: Exponentially faster language modelling

#37

Post by YetAnotherNick »

algo_trader wrote: post #8
sdrg822 wrote: post #5

Cool. Important note: """ One may ask whether the conditionality introduced by the use of CMM does not make FFFs incompatible with the processes and hardware already in place for dense matrix multiplication and deep learning more broadly. In short, the answer is “No, it does not, save for some increased caching complexity." """ It's hard to beat the hardware lottery!

Infact, as stated in the paper, this is bad news > We therefore leave the attention layers untouched Meaning, presumably, that the GPU memory remains the bottleneck Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !!

> ~$2/teraflop/s

H100 is basically ~$2/(2000 tflops/s)/hour or $1 for 4*10^18 floating point operations.

Re: Exponentially faster language modelling

#38

Post by lawlessone »

nolist_policy wrote: post #21
iNic wrote: post #10

Earlier quoted context omitted.

Do I understand correctly that the difficulty of making this useful is writing code to run this idea on GPUs?

As far as I understood it: Forget GPUs, this thing is plenty fast on CPUs. In general, GPUs are bad at branching. The fastest way to implement it on GPUs is probably to let it calculate both sides of the branch and then only use the result of the one that was taken. Which won't be faster than a normal NN.

> Forget GPUs, this thing is plenty fast on CPUs.

Does this mean everyone could be running the 100+b models from ram?

This opens up a lot , some models could be run very fast on small machines with this.

Bundling a small model inside a game to act as part of the mind for ingame NPC's (obviously with some tuning) becomes practical with this.

Re: Exponentially faster language modelling

#39

Post by lawlessone »

vorticalbox wrote: post #19

hugging face model https://huggingface.co/pbelcak/UltraFastBERT-1x11-long

> This model is provided only as sanity check for research purposes, it is untested and unfit for deployment.

I guess this means it isn't pretrained yet? Is it still just random weights?

Post reply on HN