Is there a way to do the reverse? There's a quite "legendary" Game Boy Advance game out there (Klonoa - Densetsu no Star Medal) that never got a translation to English because it has some sort of in-house created compression by Namco applied to the game that was made so it could fit into a GBA cartridge. AFAIK no one was ever able to crack it open and release the code to de/compress it. A while ago I had a "bounty" o…
Instead of trying to crack the compression algorithm directly, it would make sense to disassemble the machine code and try to understand what it does.
From the data shown it's not obvious that some of the options would ever be best, would be interesting to see if any of the non csv options ever beat the csv options and for what type of data.
From the data shown it's not obvious that some of the options would ever be best, would be interesting to see if any of the non csv options ever beat the csv options and for what type of data.
I can give you this... the larger the data the more useful parquet and compression on it will be...
Is there a way to do the reverse? There's a quite "legendary" Game Boy Advance game out there (Klonoa - Densetsu no Star Medal) that never got a translation to English because it has some sort of in-house created compression by Namco applied to the game that was made so it could fit into a GBA cartridge. AFAIK no one was ever able to crack it open and release the code to de/compress it. A while ago I had a "bounty" o…
Doesn't it depend on whether the algorithm is lossy? If it's lossy (not bijective) it's impossible to invert the function
From the data shown it's not obvious that some of the options would ever be best, would be interesting to see if any of the non csv options ever beat the csv options and for what type of data.
I can give you this... the larger the data the more useful parquet and compression on it will be...
From the brief description it seems like Parquet is just a weaker version of what a decent LZ style compressor will do anyway
Are all compressors simply being run with their default arguments? There's a lot of scope for speed/filesize tradeoffs within a single compressor.
EDIT: You are missing `csv+zstd` ? It should obsolete `csv+gzip` at all speeds and compression levels.
There is a pareto-optimality frontier here - I ran my testing back in 2016 https://code.ivysaur.me/compression-performance-test/ but the numbers are now a little bit obsolete (e.g. zstd and brotli have both seen a lot of improvements).
Me too! A few months ago I made detailed notes on how to set up your own: https://github.com/shawwn/wiki Gwern was kind enough to assist by sending over the exact version numbers of all the Haskell libraries it depends on, and answering some questions about deployment. The version numbers turned out to be crucial to getting everything running. IMO https://www.gwern.net/ is the ideal combination of style + ease of use…
Yea, look at jekyll in combination with github pages. You can see my blog for example ( https://vks.ai ), the code is hosted here: https://github.com/kootenpv/kootenpv.github.io
The downside with switching over to Jekyll is that gwern.net is increasingly dependent on Haskell integration with Pandoc, which would be difficult if you were compiling with another language: you'd have to refactor all of the additional rewrite passes (for interwiki scripts, image dimension specification, affiliate links, link annotations, inflation adjustment...) to make it at all possible, and you'd be using a lot of Haskell anyway. So it goes - the Law of Equivalent Exchange.
Is there a way to do the reverse? There's a quite "legendary" Game Boy Advance game out there (Klonoa - Densetsu no Star Medal) that never got a translation to English because it has some sort of in-house created compression by Namco applied to the game that was made so it could fit into a GBA cartridge. AFAIK no one was ever able to crack it open and release the code to de/compress it. A while ago I had a "bounty" o…
It might end up smarter to RE the decompression and then patch the game to accept an uncompressed translation (and bump up the size of the rom from 16mb to 32mb)