Show HN: Stella Nera – Maddness Hardware Accelerator
1–10 of 28 posts
Re: Show HN: Stella Nera – Maddness Hardware Accelerator
#2"Black star" in Italian. Cool name :)
Re: Show HN: Stella Nera – Maddness Hardware Accelerator
#3I am surprised that they do not mention comparing against quantized matrix multiplication because their "encoding" appears to be something like a quantization step with unevenly sized buckets. And then their approximate multiplication step to me looks like multiplying a quantized input vector against a 1-bit quantized matrix.
But overall this is an extremely exciting development because it shows how one could convert a NN into an efficient hardware implementation. And due to them working only on quantized data with LUTs, one can also embed low-dimensional matrices directly into the silicon.
My prediction would be that this will develop in the way that we can soon buy $1 hardware accelerators for things like word embedding, grammar, and general language understanding. And then you need those expensive GPUs only for the last few layers of your LLM, thereby massively reducing deployment costs.
EDIT: Reading the actual paper, I saw that this work is also related to LORA because they convert high-dimensional input vectors to a quantized value based on a lower-dimensional embedding which they call "prototypes". So it's a bit like doing LORA with 1-bit quantization but instead of representing it as 8x 1bit flags you represent it as 1x 8bit integer.
Re: Show HN: Stella Nera – Maddness Hardware Accelerator
#4Next level algorithm design with approximation of everything. I’m getting high from such proposed technology.
Re: Show HN: Stella Nera – Maddness Hardware Accelerator
#5Next level algorithm design with approximation of everything. I’m getting high from such proposed technology.
You'd be surprised how far that takes you. I mean I was truly astonished when I saw that a GptNeoX LLM quantized down to 1.5 bits per value at 90% sparsity was still producing acceptable predictions. But the size went from multiple GBs to less than 1 MB of (compressed) parameters.
Re: Show HN: Stella Nera – Maddness Hardware Accelerator
#6I am surprised that they do not mention comparing against quantized matrix multiplication because their "encoding" appears to be something like a quantization step with unevenly sized buckets. And then their approximate multiplication step to me looks like multiplying a quantized input vector against a 1-bit quantized matrix. But overall this is an extremely exciting development because it shows how one could convert…
> My prediction would be that this will develop in the way that we can soon buy $1 hardware accelerators for things like word embedding, grammar, and general language understanding. And then you need those expensive GPUs only for the last few layers of your LLM, thereby massively reducing deployment costs.
You'd still need a lot of RAM for storing these weights, wouldn't you? I mean, obviously, a $1 accelerator is a great improvement of x,000$ GPUs, but it doesn't mean we all get LLMs working on our phone just yet.
Re: Show HN: Stella Nera – Maddness Hardware Accelerator
#7I am surprised that they do not mention comparing against quantized matrix multiplication because their "encoding" appears to be something like a quantization step with unevenly sized buckets. And then their approximate multiplication step to me looks like multiplying a quantized input vector against a 1-bit quantized matrix. But overall this is an extremely exciting development because it shows how one could convert…
> My prediction would be that this will develop in the way that we can soon buy $1 hardware accelerators for things like word embedding, grammar, and general language understanding. And then you need those expensive GPUs only for the last few layers of your LLM, thereby massively reducing deployment costs. You'd still need a lot of RAM for storing these weights, wouldn't you? I mean, obviously, a $1 accelerator is a…
That's the beauty of their method: If you can replace a 8192x8192 matrix multiplication with a 8192x256 decision tree and then a 256x8192 look up table, your memory requirements go from 67,108,864 down to about 2,162,688 parameters. (I assumed that their decision tree for encoding is perfectly balanced and only uses log(256) parameters per row)
EDIT: And given that this work is centered around energy-efficiency and was sponsored by Huawei, I would guess that LLMs on your phone are precisely the goal here.
EDIT2: The process node that they did their calculations with appears to match Google's TPUv3 which has 0.56 TOPS/W and the paper claims 161 TOPS/W which would be a 280x improvement in energy efficiency over the AI chips in Pixel phones.
Re: Show HN: Stella Nera – Maddness Hardware Accelerator
#8Next level algorithm design with approximation of everything. I’m getting high from such proposed technology.
Motorola: how much is one plus one
Pentium: three
Motorola: that's incorrect!
Pentium: yeah but it's fast
Re: Show HN: Stella Nera – Maddness Hardware Accelerator
#9Earlier quoted context omitted.
> My prediction would be that this will develop in the way that we can soon buy $1 hardware accelerators for things like word embedding, grammar, and general language understanding. And then you need those expensive GPUs only for the last few layers of your LLM, thereby massively reducing deployment costs. You'd still need a lot of RAM for storing these weights, wouldn't you? I mean, obviously, a $1 accelerator is a…
That's the beauty of their method: If you can replace a 8192x8192 matrix multiplication with a 8192x256 decision tree and then a 256x8192 look up table, your memory requirements go from 67,108,864 down to about 2,162,688 parameters. (I assumed that their decision tree for encoding is perfectly balanced and only uses log(256) parameters per row) EDIT: And given that this work is centered around energy-efficiency and w…
Mind blown. Sounds almost too good to be true except the human brain runs on 20W and this brings us to the same ballpark. This was hard scifi a year ago!
Can an approach like this be integrated into stuff like llama.cpp so I could have a 200B model hashed down to 7B to run on civilian hardware or even a CPU?
Re: Show HN: Stella Nera – Maddness Hardware Accelerator
#10Ok, I get excited by seeing the numbers, but can someone please explain in a single sentence where this can be used and how big the overall impact would be?