Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators

machinelearning.apple.com

Post reply on HN
Print view View original

11–20 of 39 posts

Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators

#11

Post by timschmidt »

kingsleyopara wrote: post #10
visarga wrote: post #8

Very interesting trick, using a dictionary of basis vectors which are quickly computed from a seed without storage. But the result is the same 3 or 4 bit quantization, with only a slight improvement. Their tiles are small, just 8 or 12 weights, it's why compression doesn't go too far. It would have been great if this trick lowered quantization Congrats to Apple and Meta, makes sense they did the research, this will g…

I was about to post something similar. While the research is interesting, it doesn’t offer any advantages over 3- or 4-bit quantization. I also have to assume they explored using longer tiles but found it to be ineffective — which would make sense to me from an information theory perspective.

> it doesn’t offer any advantages over 3- or 4-bit quantization.

"zero-shot accuracy retention at 4- and 3-bit compression to be on par with or better than state-of-the-art methods, while maintaining performance comparable to FP16 baselines."

My reading of that says FP16 accuracy at Q3 or Q4 size / memory bandwidth. Which is a huge advantage.

Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators

#12

Post by kingsleyopara »

timschmidt wrote: post #11
kingsleyopara wrote: post #10

Earlier quoted context omitted.

I was about to post something similar. While the research is interesting, it doesn’t offer any advantages over 3- or 4-bit quantization. I also have to assume they explored using longer tiles but found it to be ineffective — which would make sense to me from an information theory perspective.

> it doesn’t offer any advantages over 3- or 4-bit quantization. "zero-shot accuracy retention at 4- and 3-bit compression to be on par with or better than state-of-the-art methods, while maintaining performance comparable to FP16 baselines." My reading of that says FP16 accuracy at Q3 or Q4 size / memory bandwidth. Which is a huge advantage.

For zero-shot accuracy from Table 3:

* LLaMA 3 8B: baseline 72.26, 4-bit 71.31, 3-bit 62.79

* LLaMA 3 70B: baseline 79.51, 4-bit 78.06, 3-bit 74.68

These results seem comparable to modern quantization methods—for example, the ~4-bit results for smaller LLaMA models listed here: https://ai.meta.com/blog/meta-llama-quantized-lightweight-mo...

Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators

#14

Post by timschmidt »

kingsleyopara wrote: post #12
timschmidt wrote: post #11

Earlier quoted context omitted.

> it doesn’t offer any advantages over 3- or 4-bit quantization. "zero-shot accuracy retention at 4- and 3-bit compression to be on par with or better than state-of-the-art methods, while maintaining performance comparable to FP16 baselines." My reading of that says FP16 accuracy at Q3 or Q4 size / memory bandwidth. Which is a huge advantage.

For zero-shot accuracy from Table 3: * LLaMA 3 8B: baseline 72.26, 4-bit 71.31, 3-bit 62.79 * LLaMA 3 70B: baseline 79.51, 4-bit 78.06, 3-bit 74.68 These results seem comparable to modern quantization methods—for example, the ~4-bit results for smaller LLaMA models listed here: https://ai.meta.com/blog/meta-llama-quantized-lightweight-mo...

I don't see any comparable numbers on the page you linked. Seems to only have numbers for 1B and 3B parameter models. Comparisons to AWQ and OmniQuant in Table 3 seem quite favorable with SeedLM showing 10% - 50% better performance.

Also seems like the techniques may be possible to combine.

Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators

#15

Post by pornel »

torginus wrote: post #13

This sound like compression with extra steps.. What makes this technique particular to LLM weights instead of general purpose data?

Weights in neural networks don't always need to be precise. Not all weights are equally useful to the network. There seems to be a lot of redundancy that can be replaced with approximations.

This technique seems a bit similar to lossy image compression that replaces exact pixels with a combination of pre-defined patterns (DCT in JPEG), but here the patterns aren't from cosine function, but from a pseudo-random one.

It may also be beating simple quantization from just adding noise that acts as dithering, and breaks up the bands created by combinations of quantized numbers.

Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators

#16

Post by jsenn »

torginus wrote: post #13

This sound like compression with extra steps.. What makes this technique particular to LLM weights instead of general purpose data?

> What makes this technique particular to LLM weights

This is my understanding as a non-expert.

LLM activations tend to be relatively sparse with large outliers. With linear quantization, this means you either have to clip off the outliers or you have to stretch your range to include the outliers, which wastes precious bits. Neither of these works well, so essentially all LLM quantization research is using various heuristics to get around these outliers. For example, you can do linear quantization but split the activations up into smaller blocks to make it less likely that any given block contains an outlier.

Another trick people have discovered (predates LLMs) is applying a random rotation/projection to the embeddings. This has the effect of making sure no one dimension in the vector dominates the others (which again hurts quantization). This works because in order for a single dimension to dominate, all the others have to "conspire" to be near zero. When you have 10,000+ dimensions, that's very unlikely.

This paper applies the latter trick. Instead of pre-generating the random projection matrices, they generate them on the fly on the accelerator from a seed that is fixed for each block. The seed is chosen from an offline brute-force search that needs only the weights of the network. This separates it from a lot of other quantization methods that either require calibration data or have to be simulated at training time so the network learns the quantization parameters itself.

You might think this is wasteful/might hurt performance, but it turns out that LLM inference is heavily memory-bound as it involves streaming a very large neural network into the accelerator (GPU/TPU/NPU/whatever) to operate on a relatively small amount of data, so there are lots of "free cycles" to generate these random numbers. Of course, if you care about power usage that might not be a great idea...

Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators

#17

Post by jsenn »

kingsleyopara wrote: post #10
visarga wrote: post #8

Very interesting trick, using a dictionary of basis vectors which are quickly computed from a seed without storage. But the result is the same 3 or 4 bit quantization, with only a slight improvement. Their tiles are small, just 8 or 12 weights, it's why compression doesn't go too far. It would have been great if this trick lowered quantization Congrats to Apple and Meta, makes sense they did the research, this will g…

I was about to post something similar. While the research is interesting, it doesn’t offer any advantages over 3- or 4-bit quantization. I also have to assume they explored using longer tiles but found it to be ineffective — which would make sense to me from an information theory perspective.

I think the main advantage is that you can compute the extra parameters (the PRNG seeds) from the network weights alone, whereas most other quantization methods require simulating the quantization procedure at training time (Quantization-Aware Training) or setting them from a calibration dataset (Post-Training Quantization)

Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators

#18

Post by jlcases »

This compression approach reminds me of similarities with human knowledge transfer. In both cases, we're looking for compact representations that can reconstruct complex information.

For technical documentation, I'm experimenting with a similar concept: instead of exhaustively documenting every implementation detail, defining a minimal set of principles and architectural decisions that allow "regenerating" the complete understanding.

Current LLMs excel at expanding compressed concepts, but we're still far from finding the optimal balance between explicit knowledge (detailed documentation) and implicit knowledge (patterns and principles). Is anyone working on systems applying similar ideas to technical knowledge management?

Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators

#19

Post by samus »

visarga wrote: post #8

Very interesting trick, using a dictionary of basis vectors which are quickly computed from a seed without storage. But the result is the same 3 or 4 bit quantization, with only a slight improvement. Their tiles are small, just 8 or 12 weights, it's why compression doesn't go too far. It would have been great if this trick lowered quantization Congrats to Apple and Meta, makes sense they did the research, this will g…

It should be definitely worth it because you can reuse databases of sequence to seed mappings for all future models.

Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators

#20

Post by samus »

joerick wrote: post #7
gblargg wrote: post #5

It sounds like they basically find part of a pseudo-random sequence that is closest to the desired data, then store the random seed and corrections (which are small so take less space).

Pretty fascinating from an information theory point of view. Surprising that it works at all. Is this, like, the JPEG of uniformly distributed, uncorrelated data?

We don't know. They basically look for sequences that approximate NN weights well, in the same way sinusoidal functions work well with "natural" images, but not with graphics with hard edges.
Post reply on HN