Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TurboQuant: Redefining AI efficiency with extreme compression

research.google

Post reply on HN
Print view View original

21–30 of 202 posts

Re: TurboQuant: Redefining AI efficiency with extreme compression

#21

Post by amitport »

mungoman2 wrote: post #19
amitport wrote: post #12

Earlier quoted context omitted.

r is a single value per vector. You don't have to quantize it, you can keep it and quantize the billion+ other coordinates of the vector.

What they're saying is that the error for a vector increases with r, which is true. Trivially, with r=0, the error is 0, regardless of how heavily the direction is quantized. Larger r means larger absolute error in the reconstructed vector.

Yes, the important part is that the normalized error does not increase with the dimension of the vector (which does happen when using biased quantizers)

It is expected that bigger vectors have proportionally bigger error, nothing can be done by the quantizer about that.

Re: TurboQuant: Redefining AI efficiency with extreme compression

#22

Post by valine »

maurelius2 wrote: post #14

I'm somewhat at a loss here other than understanding the fundamentals. Can someone tell me how the compression impact performance?

So let’s start with a really simple decoder transformer with a single layer and single attention head, and train it to predict the next token in a sequence of text. To predict the next token you need a few things: a query for the very last token in the sequence, and a key and value for every prior token. You take your query and compute a dot product with every prior key (two large vectors in, scaler attention score out). That scaler attention score first goes through softmax, and then becomes the weight you use to compute a weighted average of your values, new value goes through the mlp, mlp output is projected into the logits from which you sample your next token (that’s the general idea at least skipped a few steps).

The last query in the sequence will be new for every new token you predict, but the set of prior keys and values stay the same, ie keys and values are reusable. The key value cache gets bigger and bigger for each new token you add to the sequence, and that’s where compression comes in. You have to store the keys and values in vram, and you’d like to keep the size down by not storing the raw uncompressed tensors. To make this work well your compression needs two things: it needs to be fast so that you can compress and decompress on the fly, and it needs to play well with softmax attention. Prior attempts at compression usually suck at one or the other, either the speed to decompress is too slow and your token/s takes a hit, or you lose important precision and the model output quality suffers. The claim in the paper is that they’ve made progress on both.

Re: TurboQuant: Redefining AI efficiency with extreme compression

#23

Post by nvme0n1p1 »

integralid wrote: post #18
spencerflem wrote: post #6

Earlier quoted context omitted.

I think it is though- “ TurboQuant, QJL, and PolarQuant are more than just practical engineering solutions; they’re fundamental algorithmic contributions backed by strong theoretical proofs. These methods don't just work well in real-world applications; they are provably efficient and operate near theoretical lower bounds.”

I also instinctively reacted to that fragment, but at this point I think this is overreacting to a single expression. It's not just a normal thing to say in English, it's something people have been saying for a long time before LLMs existed.

There are tells all over the page:

> Redefining AI efficiency with extreme compression

"Redefine" is a favorite word of AI. Honestly no need to read further.

> the key-value cache, a high-speed "digital cheat sheet" that stores frequently used information under simple labels

No competent engineer would describe a cache as a "cheat sheet". Cheat sheets are static, but caches dynamically update during execution. Students don't rewrite their cheat sheets during the test, do they? LLMs love their inaccurate metaphors.

> QJL: The zero-overhead, 1-bit trick

> It reduces each resulting vector number to a single sign bit (+1 or -1). This algorithm essentially creates a high-speed shorthand that requires zero memory overhead.

Why does it keep emphasizing zero overhead? Why is storing a single bit a "trick?" Either there's currently an epidemic of algorithms that use more than one bit to store a bit, or the AI is shoving in extra plausible-sounding words to pad things out. You decide which is more likely.

It's 1:30am and I can't sleep, and I still regret wasting my time on this slop.

Re: TurboQuant: Redefining AI efficiency with extreme compression

#25

Post by edg5000 »

valine wrote: post #22
maurelius2 wrote: post #14

I'm somewhat at a loss here other than understanding the fundamentals. Can someone tell me how the compression impact performance?

So let’s start with a really simple decoder transformer with a single layer and single attention head, and train it to predict the next token in a sequence of text. To predict the next token you need a few things: a query for the very last token in the sequence, and a key and value for every prior token. You take your query and compute a dot product with every prior key (two large vectors in, scaler attention score o…

So limiting max context length also reduces VRAM needs a bit? If cache is 20% of total, 1/10th of context as a limit would mean 18% total memory reduction.

Re: TurboQuant: Redefining AI efficiency with extreme compression

#26

Post by pqs »

nvme0n1p1 wrote: post #23
integralid wrote: post #18

Earlier quoted context omitted.

I also instinctively reacted to that fragment, but at this point I think this is overreacting to a single expression. It's not just a normal thing to say in English, it's something people have been saying for a long time before LLMs existed.

There are tells all over the page: > Redefining AI efficiency with extreme compression "Redefine" is a favorite word of AI. Honestly no need to read further. > the key-value cache, a high-speed "digital cheat sheet" that stores frequently used information under simple labels No competent engineer would describe a cache as a "cheat sheet". Cheat sheets are static, but caches dynamically update during execution. Studen…

There is also the possibility that the article when through the hands of the company's communication department which has writers that probably write at LLM level.

Re: TurboQuant: Redefining AI efficiency with extreme compression

#27

Post by valine »

edg5000 wrote: post #25
valine wrote: post #22

Earlier quoted context omitted.

So let’s start with a really simple decoder transformer with a single layer and single attention head, and train it to predict the next token in a sequence of text. To predict the next token you need a few things: a query for the very last token in the sequence, and a key and value for every prior token. You take your query and compute a dot product with every prior key (two large vectors in, scaler attention score o…

So limiting max context length also reduces VRAM needs a bit? If cache is 20% of total, 1/10th of context as a limit would mean 18% total memory reduction.

Yup exactly, in principle it helps with both inference speed by reducing memory bandwidth usage and also reduces the memory footprint of your kvcache.

Re: TurboQuant: Redefining AI efficiency with extreme compression

#29

Post by viktorcode »

bluequbit wrote: post #2

I did not understand what polarQuant is. Is is something like pattern based compression where the algorithm finds repeating patterns and creates an index of those common symbols or numbers?

The way I understand it, it's a way of compressing vectors by switching from their per-component representation to polar coordinates representation, where the nearby vectors are clumped together to a single line, allowing to describe them by different lengths

Re: TurboQuant: Redefining AI efficiency with extreme compression

#30

Post by mskkm »

Pied Piper vibes. As far as I can tell, this algorithm is hardly compatible with modern GPU architectures. My guess is that’s why the paper reports accuracy-vs-space, but conveniently avoids reporting inference wall-clock time. The baseline numbers also look seriously underreported. “several orders of magnitude” speedups for vector search? Really? anyone has actually reproduced these results?
Post reply on HN