[dead]
Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
11–20 of 83 posts
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#12That’s so awesome! I want to try something similar. I’ve been going crazy with compression work. I reckon I can beat that prize link
Reallly?? So have you published something so far? Can i read something? Sounds like you got some interesting ideas.
I will be showcasing something on hackernews soon! Basically I found a way to “compress” a multiplayer game state from ~100KB+ to ~1KB
But it’s only for the game I’m building and it’s not pure compression work, I had to do some tricky things
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#13Earlier quoted context omitted.
Reallly?? So have you published something so far? Can i read something? Sounds like you got some interesting ideas.
I will be showcasing something on hackernews soon! Basically I found a way to “compress” a multiplayer game state from ~100KB+ to ~1KB But it’s only for the game I’m building and it’s not pure compression work, I had to do some tricky things
And just for comparison, my absolute best compression method managed to get down to 10s of KB, but the real unlock got to the ~1KB figures. Note these numbers are ALL post-compression numbers. This is not raw data vs compressed data. The ~100KB figure IS POST COMPRESSION.
For context these numbers are for a grid based game where players can perform 4 actions per second, and the numbers I’m sharing are for 30 minutes of gameplay with anywhere from 2-1024+ players (human players) playing simultaneously
So if you do the math, my compression feat is effectively ~99% compression on naive best case. And if you compare it to the raw data, it’s closing in on an even higher number than that I haven’t done the math but the raw data is another factor of 10 greater than ~100KB so the “compression” versus raw data is ~99.9%
It sounds absolutely bullshit I know :D
But I will be posting a blog post soon once I release the game.
I do compression in quotes because it’s not a pure compression feat, the 99%+ feat is effectively being clever about what actually requires compression to achieve the same outcome
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#14What does it compress the full 1GB file to? http://prize.hutter1.net/
Thanks for the link!
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#15Earlier quoted context omitted.
I will be showcasing something on hackernews soon! Basically I found a way to “compress” a multiplayer game state from ~100KB+ to ~1KB But it’s only for the game I’m building and it’s not pure compression work, I had to do some tricky things
And just for comparison, my absolute best compression method managed to get down to 10s of KB, but the real unlock got to the ~1KB figures. Note these numbers are ALL post-compression numbers. This is not raw data vs compressed data. The ~100KB figure IS POST COMPRESSION. For context these numbers are for a grid based game where players can perform 4 actions per second, and the numbers I’m sharing are for 30 minutes…
Sounds interesting man, soo am a bit confused maybe but can you run this on enwik9?
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#16Fabrice Bellard may have been the first to do this, 7 years ago: https://news.ycombinator.com/item?id=27244004
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#17What does it compress the full 1GB file to? http://prize.hutter1.net/
Maybe everyone should compress the 1st 100MB worth of digits of pi, for an apples-to-apples comparison?
Edit: oh wait that's too easy. Need to generate /publish random digits so everyone can use it.
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#18Earlier quoted context omitted.
And just for comparison, my absolute best compression method managed to get down to 10s of KB, but the real unlock got to the ~1KB figures. Note these numbers are ALL post-compression numbers. This is not raw data vs compressed data. The ~100KB figure IS POST COMPRESSION. For context these numbers are for a grid based game where players can perform 4 actions per second, and the numbers I’m sharing are for 30 minutes…
Sounds interesting man, soo am a bit confused maybe but can you run this on enwik9?
Probably not lol, it’s very specific to PvP multiplayer games, tested on my own game. But maybe I can extract the core concept to enwiki9 but I doubt it
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#19What do those compress to with conventional approaches? For comparison.
I am curious. A classic machine learning ensemble approach is to overfit a collection of small models then bag them (e.g. voting) allowing the models to generalize.
I'm sure someone's tried to overfit a bunch of transformers for compression like this, then bag them to see how well it does?
Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
#20What does it compress the full 1GB file to? http://prize.hutter1.net/
Maybe everyone should compress the 1st 100MB worth of digits of pi, for an apples-to-apples comparison? Edit: oh wait that's too easy. Need to generate /publish random digits so everyone can use it.
random digits aren't compressible though?