Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Stella Nera – Maddness Hardware Accelerator

github.com

Post reply on HN
Print view View original

21–28 of 28 posts

Re: Show HN: Stella Nera – Maddness Hardware Accelerator

#21

Post by LeanderK »

fxtentacle wrote: post #5
larodi wrote: post #4

Next level algorithm design with approximation of everything. I’m getting high from such proposed technology.

You'd be surprised how far that takes you. I mean I was truly astonished when I saw that a GptNeoX LLM quantized down to 1.5 bits per value at 90% sparsity was still producing acceptable predictions. But the size went from multiple GBs to less than 1 MB of (compressed) parameters.

do you know if the LLM was fine-tuned in any way to the sparsity & quantisation? Or did it just work out of the box?

Re: Show HN: Stella Nera – Maddness Hardware Accelerator

#22

Post by fxtentacle »

LeanderK wrote: post #21
fxtentacle wrote: post #5

Earlier quoted context omitted.

You'd be surprised how far that takes you. I mean I was truly astonished when I saw that a GptNeoX LLM quantized down to 1.5 bits per value at 90% sparsity was still producing acceptable predictions. But the size went from multiple GBs to less than 1 MB of (compressed) parameters.

do you know if the LLM was fine-tuned in any way to the sparsity & quantisation? Or did it just work out of the box?

I personally fine-tuned it with QAT = quantisation aware training and custom extensions to induce the sparsity.

https://pytorch.org/docs/stable/quantization.html#quantizati...

Re: Show HN: Stella Nera – Maddness Hardware Accelerator

#23

Post by zith »

fxtentacle wrote: post #20
maizeq wrote: post #13

Earlier quoted context omitted.

Any link to this? I actually haven't seen any reported results on less than 2 bits.

Nothing public, sorry. I do consulting on how to convert AIs from CUDA to C++ to save money. With a good quantization, you can sometimes replace a $19k A100 with a $0.5k EPYC. And especially for apps and/or WebGL interference, you want small models. Anyway, if you quantize to -1, 0, or +1 and then use arithmetic coding, you come out at around 1.58 bits per parameter. And then by skewing the distribution with forced s…

Is it possible to get good performance in computation when encoding the data this way, or is there a lot of cycles lost to packing and unpacking these bits?

Re: Show HN: Stella Nera – Maddness Hardware Accelerator

#24

Post by ddalex »

ilaksh wrote: post #12

This is a dumb question but I guess this means that you can't make something like a LoRA in software, right? Because the network is physically hardcoded?

No, the network is not hardcoded, you can build any network you want. Only the expensive step, the matmul, is hardcoded.

Re: Show HN: Stella Nera – Maddness Hardware Accelerator

#25

Post by fxtentacle »

zith wrote: post #23
fxtentacle wrote: post #20

Earlier quoted context omitted.

Nothing public, sorry. I do consulting on how to convert AIs from CUDA to C++ to save money. With a good quantization, you can sometimes replace a $19k A100 with a $0.5k EPYC. And especially for apps and/or WebGL interference, you want small models. Anyway, if you quantize to -1, 0, or +1 and then use arithmetic coding, you come out at around 1.58 bits per parameter. And then by skewing the distribution with forced s…

Is it possible to get good performance in computation when encoding the data this way, or is there a lot of cycles lost to packing and unpacking these bits?

It's actually much faster if you're limited by RAM bandwidth because instead of doing float x float mul, which requires 8 bytes of load and 4 bytes of store, you do an int8 x int8 mul with 2 bytes in and 1 byte out. And typically for a quantized LNN like this, you'd only do packing and unpacking before or after a matmul on the low-dimensional vectors so that you can directly use the quantized weights.

E.g. you quantize a 512-float activation to 512-int8, then matmul with 512x4096, Gelu, 4096x512 all in int8, then de-quantize to 512-float. That means no quantization overhead on those 4,194,304 parameters in your Dense layers.

Re: Show HN: Stella Nera – Maddness Hardware Accelerator

#26

Post by devit »

Based on the first figure in the paper, it seems that this scheme effectively turns 8 input values into a 4-bit number, thus giving an effective 0.5-bit quantization.

Considering that current aggressive quantization for LLM transformers uses 4 bits, does such a 0.5-bit quantization produce an effective neural network?

Does the scheme stay competitive if it is changed to use 4-bit quantization instead of 0.5-bit?

Re: Show HN: Stella Nera – Maddness Hardware Accelerator

#28

Post by jhj »

devit wrote: post #26

Based on the first figure in the paper, it seems that this scheme effectively turns 8 input values into a 4-bit number, thus giving an effective 0.5-bit quantization. Considering that current aggressive quantization for LLM transformers uses 4 bits, does such a 0.5-bit quantization produce an effective neural network? Does the scheme stay competitive if it is changed to use 4-bit quantization instead of 0.5-bit?

This is product quantization (a vector is chopped up into sub-vectors where each sub-vector is quantized using vector quantization (VQ)), not scalar quantization (which is what you're comparing it to here).

Also most scalar quantization methods use uniform quantization (e.g., divide the range between the scalar lower bound L and scalar upper bound H into N different regions where N is usually 2^bit_width), whereas PQ (and VQ) is learned quantization via k-means on some training vector set, so they're not really directly comparable.

Post reply on HN