Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 83 posts

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#61

Post by spidy__ »

touisteur wrote: post #57
spidy__ wrote: post #45

Earlier quoted context omitted.

Yeah yeah, I just found the idea kinda interesting so wanted to implement it

Keep exploring and writing (please ?) ! Love seeing people explore ... even after Fabrice Bellard had a go at it.

Ofcourse, am doing this just because I enjoy it.

While we are on the discussion I have mentioned a question at the end of the discussion around an assumption am trying, can you please check it out and see if you have any suggestions?

Would be awesome if someone can validate or help.

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#62

Post by spidy__ »

IncreasePosts wrote: post #60
spidy__ wrote: post #48

Earlier quoted context omitted.

1. I wrote the content as what i want to mention in the documentation and just used AI to polish it so that its easy to understand, is it hard to understand the documentation right now? 2. Have added the link for downloading both the enwik9 slice and the nyc dataset. Apologies I forgot to add it. You can get it from here - https://github.com/samyak112/pym-particles/blob/main/README.... 3. Other than zip i tested it w…

These algorithms let you specify a compression level - please note in the docs which you used. The window size can also be adjusted. Zstd might default to 4, which is "goodish compression but fast"

I tried with zstd 19

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#64

Post by spidy__ »

VorticonCmdr wrote: post #63

Great work. Just Yesterday I thought about LLMzip and asked myself if this is something which could vastly improve HTML compression when done at Google scale and shipped with browsers. I haven't done any research though.

I mean neural compressors provide great compression, BUTT the issue is they are really slow like in my project it takes around 45 minutes for de compression of 100 mb so I doubt if it would be useful, also using a transformer in user's browser sounds like a heavy task.

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#65

Post by fsiefken »

SubiculumCode wrote: post #19

What do those compress to with conventional approaches? For comparison. I am curious. A classic machine learning ensemble approach is to overfit a collection of small models then bag them (e.g. voting) allowing the models to generalize. I'm sure someone's tried to overfit a bunch of transformers for compression like this, then bag them to see how well it does?

conventional algorithms https://www.mattmahoney.net/dc/text.html

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#66

Post by pentaphobe »

spidy__ wrote: post #45
userbinator wrote: post #16

Fabrice Bellard may have been the first to do this, 7 years ago: https://news.ycombinator.com/item?id=27244004

Yeah yeah, I just found the idea kinda interesting so wanted to implement it

Hope this isnt too spicy a take, but i find it a bit disingenuous to use language that implies invention (and with no mention or citation of previous work), only to switch to dismissive language when someone notes a predecessor who you've apparently already heard of

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#67

Post by atiedebee »

spidy__ wrote: post #3
7373737373 wrote: post #2

What does it compress the full 1GB file to? http://prize.hutter1.net/

I tried it on a enwik9 100 mb slice and was able to compress it to 20 mb + 900kb transformer so 21mb. I know the top submission was able to get it to 13 mb. Still trying some ideas to get better compression.

Which slice? The large text compression benchmark uses enwik8 for a "smaller" input that is easily reproducible. The predictability of enwik9 can vary significantly depending on where in the file you are, as shown by Matt Mahoney https://www.mattmahoney.net/dc/textdata.html

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#68

Post by gcr »

gnabgib wrote: post #25
SV_BubbleTime wrote: post #21

Earlier quoted context omitted.

Random digits are compressible though. Random data does not mean it does not match a pattern in your dictionary for example.

No.. they're not. Do you understand random (the apparent or actual lack of definite patterns or predictability[0]) or compression (reduces bits by identifying and eliminating statistical redundancy[1])? [0]: https://en.wikipedia.org/wiki/Randomness [1]: https://en.wikipedia.org/wiki/Data_compression

I could write a program to generate the first 100MB of pi in a couple kilobytes. That certainly counts as “data compression” but isn’t useful outside this particular problem instance.

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#69

Post by wildstrawberry »

spidy__ wrote: post #48
wildstrawberry wrote: post #26

Three questions: 1. How much was AI used to generate documentation for this project? 2. The 100MB CSV data sources are not provided in the repo so it doesn't seem possible to reproduce your results. The enwik9 dataset says it is a "slice" of the larger data set, and there are many NYC taxi trip record datasets that exist. Can you provide the datasets used to generate your results? 3. I am surprised to see performance…

1. I wrote the content as what i want to mention in the documentation and just used AI to polish it so that its easy to understand, is it hard to understand the documentation right now? 2. Have added the link for downloading both the enwik9 slice and the nyc dataset. Apologies I forgot to add it. You can get it from here - https://github.com/samyak112/pym-particles/blob/main/README.... 3. Other than zip i tested it w…

Appreciate the followup!!!

There were a few tells of AI based on my use of AI for personal projects, especially the end section where it says "what I tried that didn't work", I've seen Claude put sections like that in the documentation.

Big thanks for linking the datasets. Here's my results from ZPAQ (max effort):

./zpaq.exe add archive.zpaq nyc_taxi_dataset_100mb_slice.txt -m5

Time: 218 seconds

Final size: 9.57MB

./zpaq.exe add archive.zpaq enwik_9_slice_100mb.txt -m5

Time: 199 seconds

Final size: 20.46MB

So, your approach is comparable to ZPAQ for the wiki dataset and achieves a better compression ratio than ZPAQ with the taxi dataset. Cool!

Bit of a tangent but if you're curious there's an interesting writeup from a few years back that compares lossless text compression algorithms at various effort levels (speed vs compression ratio). I read it recently which prompted all of my questions

https://giannirosato.com/blog/post/lossless-data-comp/

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#70

Post by spidy__ »

wildstrawberry wrote: post #69
spidy__ wrote: post #48

Earlier quoted context omitted.

1. I wrote the content as what i want to mention in the documentation and just used AI to polish it so that its easy to understand, is it hard to understand the documentation right now? 2. Have added the link for downloading both the enwik9 slice and the nyc dataset. Apologies I forgot to add it. You can get it from here - https://github.com/samyak112/pym-particles/blob/main/README.... 3. Other than zip i tested it w…

Appreciate the followup!!! There were a few tells of AI based on my use of AI for personal projects, especially the end section where it says "what I tried that didn't work", I've seen Claude put sections like that in the documentation. Big thanks for linking the datasets. Here's my results from ZPAQ (max effort): ./zpaq.exe add archive.zpaq nyc_taxi_dataset_100mb_slice.txt -m5 Time: 218 seconds Final size: 9.57MB ./…

Ohh that "what I tried that didn't work" was a section that I specifically wrote myself (and then polished with AI) because I wanted to document what are the different approaches I tried to compress more but failed.

Also thanks for the reference looks like a interesting read.

Post reply on HN