How do you reconcile this with the (I believe) widely accepted idea that you can't meaningfully compress data using offsets into Pi?
SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators
31–39 of 39 posts
Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators
#32Very interesting trick, using a dictionary of basis vectors which are quickly computed from a seed without storage. But the result is the same 3 or 4 bit quantization, with only a slight improvement. Their tiles are small, just 8 or 12 weights, it's why compression doesn't go too far. It would have been great if this trick lowered quantization Congrats to Apple and Meta, makes sense they did the research, this will g…
I was about to post something similar. While the research is interesting, it doesn’t offer any advantages over 3- or 4-bit quantization. I also have to assume they explored using longer tiles but found it to be ineffective — which would make sense to me from an information theory perspective.
This technique has three significant advantages over popular low bit quantization: 1) it retains more accuracy, 2) it does not require calibration data, 3) it's easier to implement in hardware.
Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators
#33I suspect an April fools joke. In general, compression using PRNGs is not a thing. There might be a special exception for this case, but I somewhat doubt it. =)
The version on Arxiv dates to October 2024, which likely rules this out.
I submit that there is no "compression into PRNG seeds" going on here. This is just a quantisation method that happens to leverage PRNs, which might have some specific advantages and disadvantages. What I am sure it does not do, is what it's title seems to claim, if taken literally. I suspect they're having a good laugh, getting away with what they must know is borderline trolling. I'm impressed!
Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators
#34Also from October 2024 (https://arxiv.org/abs/2410.10714)
Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators
#35How do you reconcile this with the (I believe) widely accepted idea that you can't meaningfully compress data using offsets into Pi?
Not an expert but my impression is that the title and intro are worded in a highly misleading manner.
IIUC they're transforming the data before compressing it. Also IIUC this is an established method.
Because of the nature of the data and the transform involved, you can get reasonable results with random numbers. That's already been done, but this work brute forces seeds to optimize the compression ratio and then derives the transform on the fly from the seed in order to save on memory bandwidth.
I feel like (again, non-expert) there are much deeper implications about current ML models here. The fact that a randomized transform can have this sort of impact seems to imply that there's much less information encoded by the data than we otherwise might expect given its sheer size.
Regarding Pi. You can't encode arbitrary data using arbitrary sequences and expect to come out ahead on average. But you can encode specific data using algorithms that exhibit specific behavior.
Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators
#36I suspect an April fools joke. In general, compression using PRNGs is not a thing. There might be a special exception for this case, but I somewhat doubt it. =)
I suspect that you are literally the only person on this planet who would find this to be funny enough for Apple to waste the time of a dozen AI Researchers, Meta, arXiv and Apple Legal who vet everything.
Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators
#37How do you reconcile this with the (I believe) widely accepted idea that you can't meaningfully compress data using offsets into Pi?
Not an expert but my impression is that the title and intro are worded in a highly misleading manner. IIUC they're transforming the data before compressing it. Also IIUC this is an established method. Because of the nature of the data and the transform involved, you can get reasonable results with random numbers. That's already been done, but this work brute forces seeds to optimize the compression ratio and then der…
Maybe I'm wrong. Figure 2 seems to depict exactly what's described by the title, searching for a combination of random numbers that recovers an approximation of the weights. But if that's true then I have the same question about information theoretics that you posed above.
Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators
#38What did Zuck mean that Llama 4 Behemoth is already the highest performing base model and hasnt even done training yet? What are the benchmarks then? Does he mean they did pretraining but not fine tuning?
You can fine tune a checkpoint of model during pre-training.
Re: SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators
#39How do you reconcile this with the (I believe) widely accepted idea that you can't meaningfully compress data using offsets into Pi?
You get to choose your own, more efficient "PI" for your model. Still, it's a valid question.