Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TurboQuant: A first-principles walkthrough

arkaung.github.io

Post reply on HN
Print view View original

61–70 of 78 posts

Re: TurboQuant: A first-principles walkthrough

#61

Post by [deleted] »

gajjanag wrote: post #58
amitport wrote: post #57

Earlier quoted context omitted.

Thanks for that! It is worth noting that taking advantage of the post-rotation distribution was not actually done until DRIVE (2021), which was made possible via our proper scaling. Furthermore, applying a Lloyd-Max codebook post-rotation was introduced EDEN. We consider these to be the foundational works in this regard.

> Thanks for that! It is worth noting that taking advantage of the post-rotation distribution I again feel this claim is too strong. Rotations have been used in information theory/wireless communications for decades at this point, with appropriate scaling done at channel inputs/outputs to hit channel capacity. The signals then pass through the appropriate codebooks that take advantage of the post-rotated+whitened sig…

[deleted]

Re: TurboQuant: A first-principles walkthrough

#62

Post by amitport »

gajjanag wrote: post #58
amitport wrote: post #57

Earlier quoted context omitted.

Thanks for that! It is worth noting that taking advantage of the post-rotation distribution was not actually done until DRIVE (2021), which was made possible via our proper scaling. Furthermore, applying a Lloyd-Max codebook post-rotation was introduced EDEN. We consider these to be the foundational works in this regard.

> Thanks for that! It is worth noting that taking advantage of the post-rotation distribution I again feel this claim is too strong. Rotations have been used in information theory/wireless communications for decades at this point, with appropriate scaling done at channel inputs/outputs to hit channel capacity. The signals then pass through the appropriate codebooks that take advantage of the post-rotated+whitened sig…

Thanks for the pushback, and I appreciate the reference to classical information theory.

While I probably overstated things by using the very general phrase "taking advantage," I want to be very precise about the claim, as I believe these works are foundational to quantization, beyond the scope of deep learning. The mechanism of applying a deterministic biased quantizer, such as Lloyd-Max, to the induced post-rotation distribution, alongside mathematically correcting its inherent bias, is a distinct contribution (which asymptotically improves the worst-case error).

If there is a classical paper that utilizes such a combination, I would genuinely be very eager to review it. But to my knowledge, this was not introduced prior to DRIVE and EDEN.

Re: TurboQuant: A first-principles walkthrough

#63

Post by oofbey »

kumarhn wrote: post #50
gajjanag wrote: post #45

Earlier quoted context omitted.

There are also more papers on similar themes. For example, TurboQuant makes use of QJL (quantized Johnson Lindenstrauss transformations). One of the first papers to characterize the QJL and in fact the rate distortion tradeoff for quantized matrix multiplication in general is "Optimal Quantization for Matrix Multiplication" ( https://arxiv.org/abs/2410.13780 ) by Ordentlich and Polyanskiy. There is also a more access…

TurboQuant is starting to look like a case study in how to turn a fragile paper into a breakthrough story. The attribution is thin, the “6x compression” headline is not clearly separated from prior KV-cache quantization baselines like KIVI, and the RaBitQ comparison is hard to take seriously: single-core CPU for the baseline, A100 GPU for TurboQuant. It is comparing apples-to-datacenter. Worse, there are also public…

But how can these poor googlers be expected to sift through the thousands of research papers published on these topics to find relevant citations? They don’t have time for such trivialities. They have far more important work to be doing not being evil. /s

Re: TurboQuant: A first-principles walkthrough

#64

Post by amitport »

kumarhn wrote: post #60
amitport wrote: post #53

Earlier quoted context omitted.

Those works did cite DRIVE/EDEN :) HIGGS is an extension of EDEN (using the well known method for blockwise Lloyd-Max). The proper framing of this "TurboQuant" layer in vllm (which does not include JQL) is precisely EDEN 22 without the scale correction.

EDEN is clearly relevant prior work for HIGGS. But reducing HIGGS to “an extension of EDEN” seems unfair to the authors of HIGGS. Similar primitive, different problem setting, different constraints, different contribution. Curious: where do you draw the line between “related prior work” and “an extension of EDEN”?

In the vLLM documentation quoted above, TurboQuant (which is a restricted version of EDEN) is referred to as a specific case of HIGGS. Note the symmetry: EDEN acts as a special case of HIGGS; hence, HIGGS functions as a generalization of EDEN.

In any case, the quantizer is indeed an extension, regardless of whether it was explicitly framed that way in the paper. I say this not to diminish their contribution at all, but just to clarify the relationship, as it was also stated in the vLLM doc.

Re: TurboQuant: A first-principles walkthrough

#65

Post by the_data_nerd »

linuxhansl wrote: post #2

I am fascinated by this and similar research (RotorQuant, etc). It seem by next year we will be able to run this year's largest models on last year's hardware. :) Maybe we won't need as many data centers and as much power as we thought. Maybe we can run more powerful models locally.

[dead]

Re: TurboQuant: A first-principles walkthrough

#66

Post by SilentM68 »

fragmede wrote: post #27
SilentM68 wrote: post #15

Earlier quoted context omitted.

That's my hope as well as I tend to use low end GPUs (e.g. NVIDIA GeForce RTX 2060 @ 6GB). Been looking for an image generation model that can fit that vid card, for use with Ollama + GUI in Linux. No luck yet, since money's tight and jobs are tighter :(

hey what's your Venmo?

The B-52s, er I mean Base64s:

VGhvdWdoIEkgYXBwcmVjaWF0ZSB0aGUgZ2VzdHVyZSBhbmQga2luZCBpbnRlbnRpb25zLCBJJ2QgcmF0aGVyIGxlYXJuIHRvIGZpc2ggKG9yIGNhdGNoIHRoZSBiaWcgYmFycmFjdWRhKHMpIHRoYXQgc3RvbGUgdGhlIHNjaG9vbCBvZiBmaXNoIEkgd2FzIGdpZnRlZCwgd2hpY2ggd291bGQgaGF2ZSBrZXB0IG1lIGZlZCBmb3IgbXVsdGlwbGUgbGlmZXRpbWVzLCBhbmQgc3Bvb2tlZCBhIGZldyBteXN0ZXJ5IGZyaWVuZHMgaW4gdGhlIHByb2Nlc3MtLS1hIHRhc2sgSSBhbSBjbG9zZSB0byBjb21wbGV0aW5nKSwgYW5kIG5ldmVyIGdvIGh1bmdyeSB0aGFuIGVhdCBhIGZpc2ggZm9yIGEgZGF5IGFuZCBiZSBodW5ncnkgdGhlIG5leHQu

Re: TurboQuant: A first-principles walkthrough

#68

Post by kumarhn »

amitport wrote: post #64
kumarhn wrote: post #60

Earlier quoted context omitted.

EDEN is clearly relevant prior work for HIGGS. But reducing HIGGS to “an extension of EDEN” seems unfair to the authors of HIGGS. Similar primitive, different problem setting, different constraints, different contribution. Curious: where do you draw the line between “related prior work” and “an extension of EDEN”?

In the vLLM documentation quoted above, TurboQuant (which is a restricted version of EDEN) is referred to as a specific case of HIGGS. Note the symmetry: EDEN acts as a special case of HIGGS; hence, HIGGS functions as a generalization of EDEN. In any case, the quantizer is indeed an extension, regardless of whether it was explicitly framed that way in the paper. I say this not to diminish their contribution at all, b…

[dead]

Re: TurboQuant: A first-principles walkthrough

#69

Post by krackers »

Neat thank you for creating this! That said I think someone who cares about TurboQuant probably already has a bit of linear algebra knowledge. While the initial review section is definitely appreciated, I don't think it's going to help much since you need a decent level of "mathematical maturity" anyway.

The "Coordinates of a random unit vector are all small" had me scratching my head a bit, and the language is a bit misleading since it's actually that the expected variance of any individual component is 1/N (it can't be that every coordinate is close to ±1/sqrt{N} because the mean of any individual component is clearly 0 by symmetry).

So that one should probably use more explanation since I had to work through it myself: Denoting the random unit vector {X1 ... Xn}, this is a point on a hypersphere:

* Sum[x_i ^2] = 1 (unit vector condition)

* E[Sum[X_i ^2]] = 1 (expectation of both sides)

* Sum[E[X_i ^2]] = 1 (linearity of expectation)

* E[X_1 ^2] = 1/N (by rotational symmetry E[X_1 ^2] = E[X_2 ^2] = ..)

I don't think you can make the stronger claim that E[|X_1|] = 1/sqrt{N} since that's using L1 norm on a single component, so it'd be more correct to say the RMS is just the standard deviation of the components. And this fits with the intuition that in high dimensional space has "spiky" hypercubes with the hypersphere inscribed in it close to the origin.

Re: TurboQuant: A first-principles walkthrough

#70

Post by fnordpiglet »

oofbey wrote: post #63
kumarhn wrote: post #50

Earlier quoted context omitted.

TurboQuant is starting to look like a case study in how to turn a fragile paper into a breakthrough story. The attribution is thin, the “6x compression” headline is not clearly separated from prior KV-cache quantization baselines like KIVI, and the RaBitQ comparison is hard to take seriously: single-core CPU for the baseline, A100 GPU for TurboQuant. It is comparing apples-to-datacenter. Worse, there are also public…

But how can these poor googlers be expected to sift through the thousands of research papers published on these topics to find relevant citations? They don’t have time for such trivialities. They have far more important work to be doing not being evil. /s

Gemini helped them build it but didn’t / couldn’t attribute it from its corpus. I think we will see a surge of “rediscovery” that’s unattributed training surfacing of prior work that wasn’t widely recognized at the time.
Post reply on HN