21–30 of 83 posts
Earlier quoted context omitted.
Maybe everyone should compress the 1st 100MB worth of digits of pi, for an apples-to-apples comparison? Edit: oh wait that's too easy. Need to generate /publish random digits so everyone can use it.
random digits aren't compressible though?
Random data does not mean it does not match a pattern in your dictionary for example.
Earlier quoted context omitted.
I will be showcasing something on hackernews soon! Basically I found a way to “compress” a multiplayer game state from ~100KB+ to ~1KB But it’s only for the game I’m building and it’s not pure compression work, I had to do some tricky things
And just for comparison, my absolute best compression method managed to get down to 10s of KB, but the real unlock got to the ~1KB figures. Note these numbers are ALL post-compression numbers. This is not raw data vs compressed data. The ~100KB figure IS POST COMPRESSION. For context these numbers are for a grid based game where players can perform 4 actions per second, and the numbers I’m sharing are for 30 minutes…
I started looking into diffing the state, compression, etc... until I realized, wait a minute! My player movement is linear so I only need a packet for start and stop! And so I achieved near infinite efficiency improvement :)
I think the word is... a specialized solution can beat a general one.
Also, "remembering what the program actually needs to do, and just making it do that"... I de-pessimized the netcode: https://youtube.com/watch?v=pgoetgxecw8
What does it compress the full 1GB file to? http://prize.hutter1.net/
I tried it on a enwik9 100 mb slice and was able to compress it to 20 mb + 900kb transformer so 21mb. I know the top submission was able to get it to 13 mb. Still trying some ideas to get better compression.
Earlier quoted context omitted.
random digits aren't compressible though?
Random digits are compressible though. Random data does not mean it does not match a pattern in your dictionary for example.
1. How much was AI used to generate documentation for this project?
2. The 100MB CSV data sources are not provided in the repo so it doesn't seem possible to reproduce your results. The enwik9 dataset says it is a "slice" of the larger data set, and there are many NYC taxi trip record datasets that exist. Can you provide the datasets used to generate your results?
3. I am surprised to see performance comparisons only between your transformer and WinZIP. What were your results when comparing your transformer to more modern approaches like LZMA2 (level 9), BZIP2 and ZPAQ (max effort)?
Earlier quoted context omitted.
Random digits are compressible though. Random data does not mean it does not match a pattern in your dictionary for example.
No.. they're not. Do you understand random (the apparent or actual lack of definite patterns or predictability[0]) or compression (reduces bits by identifying and eliminating statistical redundancy[1])? [0]: https://en.wikipedia.org/wiki/Randomness [1]: https://en.wikipedia.org/wiki/Data_compression
Earlier quoted context omitted.
Random digits are compressible though. Random data does not mean it does not match a pattern in your dictionary for example.
No.. they're not. Do you understand random (the apparent or actual lack of definite patterns or predictability[0]) or compression (reduces bits by identifying and eliminating statistical redundancy[1])? [0]: https://en.wikipedia.org/wiki/Randomness [1]: https://en.wikipedia.org/wiki/Data_compression