Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TurboQuant: A first-principles walkthrough

arkaung.github.io

Post reply on HN
Print view View original

11–20 of 78 posts

Re: TurboQuant: A first-principles walkthrough

#11

Post by linuxhansl »

everythingctl wrote: post #4
linuxhansl wrote: post #2

I am fascinated by this and similar research (RotorQuant, etc). It seem by next year we will be able to run this year's largest models on last year's hardware. :) Maybe we won't need as many data centers and as much power as we thought. Maybe we can run more powerful models locally.

Maybe we can run more powerful models locally. I thought the principal consequence of these KV cache optimisations was letting you run more simultaneous inferences on the same model with the same memory. It doesn’t let you store more model. In some sense that puts local LLM usage at a further disadvantage to inference done in a hyperscaler’s data center.

The size of the KV cache (context stored) is proportional to the number of layers of the model and number of "hidden dimensions". For a 400B model it could be 30-60GB for just an 8K context window (depends on the model, etc, just a ballpark).

So shrinking that by 6x (from fp16), would be big win for larger models. True, while TurboQuant can also be applied to model weights, it won't save size over q4 compression, but will have better accuracy.

Edits: Better context

Re: TurboQuant: A first-principles walkthrough

#12

Post by amitport »

theredsix wrote: post #10
amitport wrote: post #6

TurboQuant is a restricted version of EDEN quantization (NeurIPS 21, ICML 22). It lacks the optimal scale derivations, which makes the TurboQuant variant considerably less accurate than those works. We show this thoroughly in a new note at https://arxiv.org/abs/2604.18555 . We were the first to introduce post-rotation distribution-aware quantization in 2021. This was later implemented in many fields, including federa…

Are you guys going to follow up with a paper showing EDEN results match or beat turboquant for needle in a haystack benchmarks?

The note includes extensive experiments and reproduces many of the figures from the TurboQuant paper in our Section 5. Honestly, I think our case is pretty clear-cut as is. I am not sure what the overhead for those specific benchmarks would be, but we will look into it.

(In any case, I want to emphasize that TurboQuant quantizer is a private case of EDEN)

Re: TurboQuant: A first-principles walkthrough

#13

Post by qingcharles »

linuxhansl wrote: post #2

I am fascinated by this and similar research (RotorQuant, etc). It seem by next year we will be able to run this year's largest models on last year's hardware. :) Maybe we won't need as many data centers and as much power as we thought. Maybe we can run more powerful models locally.

We're only a few years into this new tech getting serious research manhours thrown at it. Already some incredible optimizations have been found in a short amount of time. Not only has the efficiency of inference been increasing dramatically, the quality of tiny models has been significantly improving.

The future is bright for local AI.

Re: TurboQuant: A first-principles walkthrough

#15

Post by SilentM68 »

everythingctl wrote: post #4
linuxhansl wrote: post #2

I am fascinated by this and similar research (RotorQuant, etc). It seem by next year we will be able to run this year's largest models on last year's hardware. :) Maybe we won't need as many data centers and as much power as we thought. Maybe we can run more powerful models locally.

Maybe we can run more powerful models locally. I thought the principal consequence of these KV cache optimisations was letting you run more simultaneous inferences on the same model with the same memory. It doesn’t let you store more model. In some sense that puts local LLM usage at a further disadvantage to inference done in a hyperscaler’s data center.

That's my hope as well as I tend to use low end GPUs (e.g. NVIDIA GeForce RTX 2060 @ 6GB). Been looking for an image generation model that can fit that vid card, for use with Ollama + GUI in Linux. No luck yet, since money's tight and jobs are tighter :(

Re: TurboQuant: A first-principles walkthrough

#19

Post by meehai »

amitport wrote: post #12
theredsix wrote: post #10

Earlier quoted context omitted.

Are you guys going to follow up with a paper showing EDEN results match or beat turboquant for needle in a haystack benchmarks?

The note includes extensive experiments and reproduces many of the figures from the TurboQuant paper in our Section 5. Honestly, I think our case is pretty clear-cut as is. I am not sure what the overhead for those specific benchmarks would be, but we will look into it. (In any case, I want to emphasize that TurboQuant quantizer is a private case of EDEN)

with the amount of traction this has gotten... coming with a clear set of experiments even on arxiv paper would be of great help to showcase your improvements. And if they're easily reproducible, they could get integrated in the mainstream inference engines as well, as the main point here is compression with little degradation.
Post reply on HN