[flagged]
LLM slop. See their other comment which is even more obvious.
[dead]
TurboQuant: Redefining AI efficiency with extreme compression
171–180 of 202 posts
Re: TurboQuant: Redefining AI efficiency with extreme compression
#172Re: TurboQuant: Redefining AI efficiency with extreme compression
#173Someone implementing it on llamacpp already https://github.com/mudler/llama.cpp/commit/dee102db1bfd723c9...
[dead]
Re: TurboQuant: Redefining AI efficiency with extreme compression
#174Re: TurboQuant: Redefining AI efficiency with extreme compression
#175Re: TurboQuant: Redefining AI efficiency with extreme compression
#176This is a great development for KV cache compression. I did notice a missing citation in the related works regarding the core mathematical mechanism, though. The foundational technique of applying a geometric rotation prior to extreme quantization, specifically for managing the high-dimensional geometry and enabling proper bias correction, was introduced in our NeurIPS 2021 paper, "DRIVE" ( https://proceedings.neurip…
LOL. This is a classical technique, Johnson-Linderstrauss etc. In this context, rediscovered every few years (recently months), e.g. here's 2017: https://proceedings.mlr.press/v70/suresh17a
We do mention and the paper you shared. Please read our paper to see how the rotation-aware bias correction we introduced efficiently fixes the bias and provides a better worst-case error.
Re: TurboQuant: Redefining AI efficiency with extreme compression
#177Earlier quoted context omitted.
I like the visualization, but I don’t understand the grid quantization. If every point is on the unit circle aren’t all the center grid cords unused?
Yes. Great catch. I simplified the grid just for visualization purpose. I've updated the visualization. The grid is actually not uniformly spaced. Each coordinate is quantized independently using optimal centroids for the known coordinate distribution. In 2D, unit-circle coordinates follow the arcsine distribution (concentrating near ±1), so the centroids cluster at the edges, not the center.
Cool! Thank you
Re: TurboQuant: Redefining AI efficiency with extreme compression
#178Earlier quoted context omitted.
Someone else linked that elsewhere in the comments and while it's certainly a nice visual it seems like it's not accurately portraying the paper. Isn't the grid supposed to have a weird alignment that depends on the bit depth? And there's supposed to be a second quantization step involving the residual.
Fair point. I've updated the animation to address this. The grid now uses the correct non-uniform centroids (optimal for the arcsine distribution in 2D), so you'll see grid lines cluster near the edges where unit-circle coordinates actually concentrate, rather than being evenly spaced. The spacing does change with bit depth. On the second quantization step: the paper's inner-product variant uses (b-1) bits for the MS…
It looks nice! Fair enough about QJL - it seems to be nothing more than an unbiasing measure anyway.
I'm not sure if it's my own misunderstanding or if the paper [0] has something of an error. Section 3.1 starts out to the effect "let x be on the unit hypersphere" (but I'm fairly certain it's actually not). Neither algorithm 1 nor algorithm 2 show a normalization step prior to rotating x. Algorithm 2 line 8 shows that the scalar returned is actually the magnitude of the residual without accounting for QJL.
Anyway I'm pretty sure the authors inadvertently omitted that detail which really had me confused for a while there.
Re: TurboQuant: Redefining AI efficiency with extreme compression
#179Here's my attempt at a undergrad-level summary (corrections welcome!): The core idea is to quantize KV cache, but do so in a way that destroys minimal information. In this case, it's similarly scores between vectors. The simplest way to do this is to change all the elements from 16bit of precision to, say, 4 bits (Scalar Quant.). These papers improve on it by realizing: almost all the energy (concentration of measure…
Some corrections: the vectors are un-rotated in practice for future query vectors. This could be removed with a slightly different LLM arch. PolarQuant does live on in TurboQuant's codebooks for quantization which borrows from the hyperpolar coords
> added in some gross / highly-pragmatic extra bits for important channels
I'm curious what you meant by that. I understood it to only have the MSE quantization vector, a 1-bit QJL vector, and a scalar magnitude.
> PolarQuant does live on in TurboQuant's codebooks for quantization which borrows from the hyperpolar coords
Isn't the turbo codebook the irregularly spaced centroid grid?
Re: TurboQuant: Redefining AI efficiency with extreme compression
#180Earlier quoted context omitted.
The whole goal of quantisation is to put the data into 'bins' so that it can easily be 'packed' so that you can represent it using less bits (less information). You can think of it like rounding essentially (3.14159 -> 3). Now, sometimes within data, the distribution will be non-ideal for separating it out into bins (let's say that our rounding rules are simple -- we simply use a floor function so 2.45 maps to 2 and…
Amazing explanation! Thank you so much for taking the time to put it together. It makes a lot of sense. I’m not the one who asked the question, but I was impressed by such eloquent and clearly explained answer
Thank you! I'm glad you found it helpful (and that others did too)!!