So has anyone tried to you know for example keep constant weights base model and just transmit the data, might be better compression
Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
41–50 of 83 posts
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#42What does it compress the full 1GB file to? http://prize.hutter1.net/
Maybe everyone should compress the 1st 100MB worth of digits of pi, for an apples-to-apples comparison? Edit: oh wait that's too easy. Need to generate /publish random digits so everyone can use it.
Compressor: Output an empty file.
Decompressor: Take any old algorithm for finding digets of pi, find first 100M of them, print them.
Compression ratio of 0! :0
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#43if first bit is 1 then decompress to a picture of my cat, else its just Huffman
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#44Somewhat related is stavros's method to compress 500KB to something like 50 bytes https://www.stavros.io/posts/compressing-images-with-stable-...
main drawback is that it's not lossless ;-)
but this is great. I hope this actually becomes a format that wraps the weights and transformer module (maybe this can also be NAS-optimized too?). Maybe it would even work for video?
It's like calling gzip but instead of compression level you choose kolmogorov complexity level
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#45Fabrice Bellard may have been the first to do this, 7 years ago: https://news.ycombinator.com/item?id=27244004
Yeah yeah, I just found the idea kinda interesting so wanted to implement it
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#46So has anyone tried to you know for example keep constant weights base model and just transmit the data, might be better compression
I might be confused by the question, but I overfit the model on a single file and then transport the model along with the arithmetic coding file. There have been ideas where you generalize a model (constant weights) and then pass the arithmetic coding file along with it. So that way you only pass the arithmetic coding file.
BUT my model size is just 900KB (for 100mb file atleast) so it is negligible
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#47Dumb question: can you train a model to predict the next byte of ANOTHER MODEL So apply this same logic to compressing a bigger model within a smaller model I know this is absolutely regarded, but humour me please
If there's any redundancy in the model that can be compressed (parallel to how RLE is used to compress the static Huffman tree in FLATE) that's possible, but it's not necessary if the model is being trained on the input dynamically, like what Bellard's NNCP does.
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#48Three questions: 1. How much was AI used to generate documentation for this project? 2. The 100MB CSV data sources are not provided in the repo so it doesn't seem possible to reproduce your results. The enwik9 dataset says it is a "slice" of the larger data set, and there are many NYC taxi trip record datasets that exist. Can you provide the datasets used to generate your results? 3. I am surprised to see performance…
1. I wrote the content as what i want to mention in the documentation and just used AI to polish it so that its easy to understand, is it hard to understand the documentation right now?
2. Have added the link for downloading both the enwik9 slice and the nyc dataset. Apologies I forgot to add it.
You can get it from here - https://github.com/samyak112/pym-particles/blob/main/README....
3. Other than zip i tested it with zstd19, and now that you mentioned LZMA2 and BZIP2
I got results on enwik9 100mb slice as
zstd - 28mb bzip2 - 30mb lzma2 - 26mb
I will mention these and results from ZPAQ in the readme for both files, thanks for pointing them out!!!
But the thing is this neural compression approach cant be used right now, as it takes hours to compress and de compress a 100mb file so not really usable and more of a fun project.
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#49I've had this idea of building a codec that would similarly overfit to specific images. But the codec itself would not be a fixed size transformer... instead you could just mess around with the sizing to get better quality/smaller size. So the codec would be something like: I've seen experiments where people have a "fixed" pipeline but I think having something more dynamic would work quite well.
Likely doable with metaparameter tuning (used to work on a team with data scientists that were routinely doing this in various situations). Seems like a cool idea.
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#50Earlier quoted context omitted.
I tried it on a enwik9 100 mb slice and was able to compress it to 20 mb + 900kb transformer so 21mb. I know the top submission was able to get it to 13 mb. Still trying some ideas to get better compression.
Since you know the size of the file beforehand you may be able to overfit some kind of text diffusion model instead of a transformer? May allow you to partially correct the model output using some other method and then fill in the blanks that were wrong from previous generations.
Oh, sounds interesting. I hadn't considered using a diffusion model for this. My current approach generates the document byte by byte with an autoregressive transformer, so I'm curious how a diffusion model would improve memorization or reconstruction quality.
Can you point me to something that i can read? I really wanna try this approach , diffusion model does sounds interesting for compression.