Using machine learning to choose compression algorithms
1–10 of 60 posts
Re: Using machine learning to choose compression algorithms
#2Isn't one of the benefits of time-series databases more compact storage at minimal overhead?
Re: Using machine learning to choose compression algorithms
#3Is there a way to do the reverse?
There's a quite "legendary" Game Boy Advance game out there (Klonoa - Densetsu no Star Medal) that never got a translation to English because it has some sort of in-house created compression by Namco applied to the game that was made so it could fit into a GBA cartridge. AFAIK no one was ever able to crack it open and release the code to de/compress it.
A while ago I had a "bounty" of USD100 for anyone that could do it (just the decompression and re-compression, not translation) but there aren't many people that want to fiddle with low-level GBA coding.
Re: Using machine learning to choose compression algorithms
#4nice project. the next step is to use ML to create better compression algorithms
Re: Using machine learning to choose compression algorithms
#5Isn't one of the benefits of time-series databases more compact storage at minimal overhead?
Definitely! I really optimized for "no development time spent" and was just using pandas to extract html tables into csv and just store them :-). 2 lines of code really. I had no idea I would have it running for so long.
It was really just the example that made me wonder why I have to consider which compression would be best for files with my characteristics - but not saying it was best practice to begin with haha!
Re: Using machine learning to choose compression algorithms
#6Isn't one of the benefits of time-series databases more compact storage at minimal overhead?
I'm a long time fan of your blog :O
Re: Using machine learning to choose compression algorithms
#7How does it compare to context mixing algorithms such as https://en.wikipedia.org/wiki/ZPAQ ?
Re: Using machine learning to choose compression algorithms
#8Is there a way to do the reverse? There's a quite "legendary" Game Boy Advance game out there (Klonoa - Densetsu no Star Medal) that never got a translation to English because it has some sort of in-house created compression by Namco applied to the game that was made so it could fit into a GBA cartridge. AFAIK no one was ever able to crack it open and release the code to de/compress it. A while ago I had a "bounty" o…
I'm assuming the game is playable, i.e. the decompression code is included on the cartridge and you just don't know how it works. In that case you could emulate the game and use a language model to identify strings containing Japanese text (you'd need to know the encoding to do that) so they can be extracted for translation. That doesn't allow you to put the translations into the compressed code, but you might be able to instrument the emulator to inject translated strings on-the-fly.
Re: Using machine learning to choose compression algorithms
#9How does it compare to context mixing algorithms such as https://en.wikipedia.org/wiki/ZPAQ ?
shrynk is not doing any compression itself, it's using ML to do a "meta" approach: it computes features on your data like how many rows, how many columns, how much duplication etc, and then predicts which existing compression algorithm (out of the available compressions) will be best given your requirements/preferences in terms of size, write time and read time.
Re: Using machine learning to choose compression algorithms
#10A somewhat different example but related (at least by name) but I recall an article from Chris Wellons' blog, Null Program, where he wanted to "discover" a new hashing algorithm, so he randomly generated them, JT compiled them to native, then tested them.
There was, how ever, no machine learning or optimizing. Instead, he called it "prospecting" and just generate a new one from scratch each time until he found something interesting.