We did a fun experiment in Chapter 5 of Classic Computer Science Problems in Python. We ran a genetic algorithm to find what order of a list (assuming the order doesn't matter) of names led to the best compression using one of the Python standard library's built-in compression algorithms. This article has made me interested in re-running the experiment with multiple compression algorithms.
Using machine learning to choose compression algorithms
41–50 of 60 posts
Re: Using machine learning to choose compression algorithms
#42Is it possible to run data through a DNN, making sure the output is the same as the input (or close for lossless data), then take an autoencoder variant, record the 'compressed' data, then on the other end have the other half of the DNN to decompress it? I'm pretty ignorant on the topic, so I get that that may be off, but if so, why wouldn't that be a valid solution to compressing data?
Sure, and you can add some correction bits to fix the lossy encoding of the autoencoder.
The problem is that you have to encode/send the DNN itself, otherwise your receiver won't know how to decode the data. If you are not smart, the added codelength of the DNN will likely blow away your savings. If you are smart, this leads to a whole formulation of machine learning called MDL:
Re: Using machine learning to choose compression algorithms
#43Why not use ML to build a statistical model of your data, and then use that model to compress it? You will likely get better codelengths for your specific dataset than off-the-shelf algorithms can achieve. And if you figure out a good model that describes the fluctuations in the cryptocurrency prices very well, then you can use that model pretty directly to make money via automated trading.
I've got an easy-to-use library for arithmetic encoding in Java, it would be easy to port to other languages: https://github.com/comperical/MirrorEncode
Re: Using machine learning to choose compression algorithms
#44Isn't one of the benefits of time-series databases more compact storage at minimal overhead?
Does anyone know what is the leading time-series database people use today? Like, eg, bar / tick data. (I use PostgreSQL, due to my ignorance.)
I've been happy with influxdb, but I've also noticed that a lot of people switch to a time series database way before they need to. You can go pretty far with postgres/oracle/sql server, and it has the advantage you don't need to manage different databases.
Re: Using machine learning to choose compression algorithms
#45Are all compressors simply being run with their default arguments? There's a lot of scope for speed/filesize tradeoffs within a single compressor. EDIT: You are missing `csv+zstd` ? It should obsolete `csv+gzip` at all speeds and compression levels. There is a pareto-optimality frontier here - I ran my testing back in 2016 https://code.ivysaur.me/compression-performance-test/ but the numbers are now a little bit obso…
+1 - and if you could use zstd with custom dictionary, then you can achieve even better compression ratios
Re: Using machine learning to choose compression algorithms
#46Earlier quoted context omitted.
You might as well write a tool that extracts strings from a video signal using OCR, and translates them. That would make the solution more universal, and you could even use it to e.g. suppress ads.
Well that's super hard since Japanese encoding is an epic story in digital archaeology itself.
I'm not well-versed on the subject, how does encoding come into play for text displayed on the screen? Did they use a strange way of representing the Japanese text because of technical limitations?
Re: Using machine learning to choose compression algorithms
#47Is there a way to do the reverse? There's a quite "legendary" Game Boy Advance game out there (Klonoa - Densetsu no Star Medal) that never got a translation to English because it has some sort of in-house created compression by Namco applied to the game that was made so it could fit into a GBA cartridge. AFAIK no one was ever able to crack it open and release the code to de/compress it. A while ago I had a "bounty" o…
Doesn't it depend on whether the algorithm is lossy? If it's lossy (not bijective) it's impossible to invert the function
This is discussing text or code compression. There's no point in using lossy methods for the dialogue of your game.
Re: Using machine learning to choose compression algorithms
#48Earlier quoted context omitted.
I'm assuming the game is playable, i.e. the decompression code is included on the cartridge and you just don't know how it works. In that case you could emulate the game and use a language model to identify strings containing Japanese text (you'd need to know the encoding to do that) so they can be extracted for translation. That doesn't allow you to put the translations into the compressed code, but you might be abl…
The GBA didn't have much RAM. There is a good chance tiny chunks of the game get decompressed as needed, and there is never a time when the whole thing is decompressed at once and can be dumped.
You don't need everything to be decompressed all at once to dump it. You could continuously dump the memory contents while exploring the game (or having a fuzzer do it for you).
Re: Using machine learning to choose compression algorithms
#49Are all compressors simply being run with their default arguments? There's a lot of scope for speed/filesize tradeoffs within a single compressor. EDIT: You are missing `csv+zstd` ? It should obsolete `csv+gzip` at all speeds and compression levels. There is a pareto-optimality frontier here - I ran my testing back in 2016 https://code.ivysaur.me/compression-performance-test/ but the numbers are now a little bit obso…
+1 - and if you could use zstd with custom dictionary, then you can achieve even better compression ratios
Yea zstd is really amazing... if I would choose a single one all the time it'd be zstd for sure.
Re: Using machine learning to choose compression algorithms
#50Are all compressors simply being run with their default arguments? There's a lot of scope for speed/filesize tradeoffs within a single compressor. EDIT: You are missing `csv+zstd` ? It should obsolete `csv+gzip` at all speeds and compression levels. There is a pareto-optimality frontier here - I ran my testing back in 2016 https://code.ivysaur.me/compression-performance-test/ but the numbers are now a little bit obso…
Yea, I thought parameters per compression algorithm should indeed be added in a next version :) more compute but definitely an improvement.
I think pandas doesn't offer zstd as option with csv, but I'll check once more.
EDIT: indeed, it's missing in `to_csv` - seems like an oversight.