What's the patent encumberment status of this algorithm?
OpenZL: An open source format-aware compression framework
81–90 of 110 posts
Re: OpenZL: An open source format-aware compression framework
#82One of the mentioned examples sounds like the compressor is taking advantage of the SDDL by treating row-oriented data as stripes of column-oriented data, and then compressing that. This makes me curious - for data that’s already column-oriented like Parquet, what’s the advantage of OpenZL over zstd?
Re: OpenZL: An open source format-aware compression framework
#83I've recently been wondering: could you re-compress gzip to a better compression format, while keeping all instructions that would let you recover a byte-exact copy of the original file? I often work with huge gzip files and they're a pain to work with, because decompression is slow even with zlib-ng.
That's called `pristine-gz`, part of the `pristine-tar` project.
Thank you! It seems to be what I'm looking for.
Re: OpenZL: An open source format-aware compression framework
#84I've recently been wondering: could you re-compress gzip to a better compression format, while keeping all instructions that would let you recover a byte-exact copy of the original file? I often work with huge gzip files and they're a pain to work with, because decompression is slow even with zlib-ng.
precomp/antix/... are tools that can bruteforce the original gzip parameters and let you recreate the byte-identical gzip archive. The output is something like {precomp header}{gzip parameters}{original uncompressed data} which you can then feed to a stronger compressor. A major use case is if you have a lot of individually gzipped archives with similar internal content, you can precomp them and then use long-range s…
> A major use case is if you have a lot of individually gzipped archives with similar internal content, you can precomp them and then use long-range solid compression over all your archives together for massive space savings.
Or even a single gzipped archive with similar pieces of content that are more than 32KB apart.
Re: OpenZL: An open source format-aware compression framework
#85Earlier quoted context omitted.
Thanks. I thought basis also had specific encoders depending on the typical average / nature of the data input, like this OpenZL project
It probably does have different modes that it selects based on the input data. I don't know that much about the implementation of image compression, but I know that PNG for example has several preprocessing modes that can be selected based on the image contents, which transform the data before entropy encoding for better results. The difference with OpenZL IIUC seems to be that it has some language that can flexibly…
Thank you for the explanation !
Re: OpenZL: An open source format-aware compression framework
#86One of the mentioned examples sounds like the compressor is taking advantage of the SDDL by treating row-oriented data as stripes of column-oriented data, and then compressing that. This makes me curious - for data that’s already column-oriented like Parquet, what’s the advantage of OpenZL over zstd?
[deleted]
Re: OpenZL: An open source format-aware compression framework
#87Plans for language bindings? Should be trivial to whip up simpler ones like python or dotnet but I didn't see any official bindings yet.
Re: OpenZL: An open source format-aware compression framework
#88Earlier quoted context omitted.
Ooh, thanks for mentioning these! I wasn't aware of the existence of these tools but yes it seems very possible that you could transform these other spec formats into SDDL descriptions. I'll check them out.
There are a ton of these. GNU Poke comes to mind too.
Here's a list: https://github.com/dloss/binary-parsing
Re: OpenZL: An open source format-aware compression framework
#89it reminds me of the EXI compression for XML that can be very optimized with a XSD Schema with a schema aware compression, that also use the schema graph for optimal compression :
https://www.w3.org/TR/exi-primer/
Re: OpenZL: An open source format-aware compression framework
#90Wow this sounds nuts. I want to try this on some large csvs later today.
I must be doing something wrong but I couldn't manage to compressed a file using a custom trained profile since I was getting this error:
``` src/openzl/codecs/dispatch_string/encode_dispatch_string_binding.c:74: EI_dispatch_string: splitting 48000001 strings into 14 outputs OpenZL Library Exception: OpenZL error code: 55 OpenZL error string: Input does not respect conditions for this node OpenZL error context: Code: Input does not respect conditions for this node Message: Check `eltWidth != 2' failed where: lhs = (unsigned long) 4 rhs = (unsigned long) 2
Graph ID: 5 Stack Trace: #0 doEntropyConversion (src/openzl/codecs/entropy/encode_entropy_binding.c:788): Check `eltWidth != 2' failed where: lhs = (unsigned long) 4 rhs = (unsigned long) 2
#1 EI_entropyDynamicGraph (src/openzl/codecs/entropy/encode_entropy_binding.c:860): Forwarding error: #2 CCTX_runGraph_internal (src/openzl/compress/cctx.c:770): Forwarding error: #3 CCTX_runSuccessor_internal (src/openzl/compress/cctx.c:1149): Forwarding error: #4 CCTX_runSuccessors (src/openzl/compress/cctx.c:707): Forwarding error: #5 CCTX_runSuccessor_internal (src/openzl/compress/cctx.c:1149): Forwarding error: #6 CCTX_runSuccessors (src/openzl/compress/cctx.c:707): Forwarding error: #7 CCTX_runSuccessor_internal (src/openzl/compress/cctx.c:1149): Forwarding error: #8 CCTX_runSuccessors (src/openzl/compress/cctx.c:707): Forwarding error: #9 CCTX_runSuccessor_internal (src/openzl/compress/cctx.c:1149): Forwarding error: #10 CCTX_runSuccessors (src/openzl/compress/cctx.c:707): Forwarding error: #11 CCTX_runSuccessor_internal (src/openzl/compress/cctx.c:1149): Forwarding error: #12 CCTX_runSuccessors (src/openzl/compress/cctx.c:707): Forwarding error: #13 CCTX_runSuccessor_internal (src/openzl/compress/cctx.c:1149): Forwarding error: #14 CCTX_startCompression (src/openzl/compress/cctx.c:1276): Forwarding error: #15 CCTX_compressInputs_withGraphSet_stage2 (src/openzl/compress/compress2.c:116): Forwarding error: ```
On the other hand the default CSV profile didn't seem that great either, the CSV file was 349 MB and it compressed it down to 119MB while a ZIP file of the CSV is 105MB.