Another library in this space is pcodec; I'd appreciate a comparison of the two.
Agreed; pcodec is probably one of the most relevant comparisons. I will add pcodec to teh benchmark
Fc, a lossless compressor for floating-point streams
21–30 of 35 posts
Re: Fc, a lossless compressor for floating-point streams
#22Earlier quoted context omitted.
It is intended t obe mainly source agnostic (will try to add custom source predictors too). The idea is to treat input as an ordered stream of doubles and look for numeric structure like repeats, smooth deltas, fixed increments, or low-entropy bits. Target presentlyis scientific/time-series/simulation/analytics data, not photos or sound.
isn't sound a time series? I guess it's not usually 64-bit doubles.
Yes it is. The mismatch is mainly representation and purpose: audio is usually int16/int24/float32 PCM, and audio codecs often exploit perceptual loss. fc is lossless and currently tuned for float64 streams.
Re: Fc, a lossless compressor for floating-point streams
#23I built "fc", a C library for compressing streams of 64-bit floating-point values without quantization. It is not trying to replace zstd or lz4. The idea is narrower: take blocks of doubles, try a set of float-specific predictors/transforms/coders, and emit whichever representation is smallest for that block. It is aimed at time-series, scientific, simulation, and analytics data where the numbers often have structure…
> rather than production-hardened.
Please run it through your preferred AI once or twice with instruction to look for bugs. The version of Fc in the main branch has at least a few memory safety bugs that attacker-controlled inputs could exploit.
I'd link a chat history but the tool I used has that feature blocked for some weird reason, and the locals round these parts don't take kindly to copy-pasted AI content...
Re: Fc, a lossless compressor for floating-point streams
#24The question is, how close can OpenLZ come? (This is from the same people who develop zstd, but suitable for structured data in a generic way.)
I need to add it to the benchmark. My expectation is that OpenZL should be strong when the enclosing format is known and SDDL can separate typed fields cleanly. Running both on the same f64 arrays will give some information
Re: Fc, a lossless compressor for floating-point streams
#25I built "fc", a C library for compressing streams of 64-bit floating-point values without quantization. It is not trying to replace zstd or lz4. The idea is narrower: take blocks of doubles, try a set of float-specific predictors/transforms/coders, and emit whichever representation is smallest for that block. It is aimed at time-series, scientific, simulation, and analytics data where the numbers often have structure…
> rather than production-hardened. Please run it through your preferred AI once or twice with instruction to look for bugs. The version of Fc in the main branch has at least a few memory safety bugs that attacker-controlled inputs could exploit. I'd link a chat history but the tool I used has that feature blocked for some weird reason, and the locals round these parts don't take kindly to copy-pasted AI content...
Thank you. Fuzz safety is definitely on my list. Current focus is to broaden the benchmarks , predictors and preprocessors and see what sticks
Re: Fc, a lossless compressor for floating-point streams
#26A lossy compressor might also be useful for common floating point apps. The simplest compressor ever would just chop off a number of bits from the mantissa.
Re: Fc, a lossless compressor for floating-point streams
#27A lossy compressor might also be useful for common floating point apps. The simplest compressor ever would just chop off a number of bits from the mantissa.
Yeah, and also approximating a double (within range) to int32 :)
Re: Fc, a lossless compressor for floating-point streams
#28Those interested in this might find my paper on "Representing numeric data in 32 bits while preserving 64-bit precision" to be of interest. Can be found at
https://arxiv.org/abs/1504.02914 (note the code available as auxilliary files). In the context of this compressor, it could be one of the compressors competing to compress a block. It works well for data converted from a decimal representation with a small number of digits.
Re: Fc, a lossless compressor for floating-point streams
#29I must say, for a library advertising handling of streams of data, the absence of a stream utility to [input] | fc | fc -d surprised me.
I understand this is more the primitive that you would build such a thing on top of, just that the first question I always have for novel compressors is "how do they do on these example streams of data".
Re: Fc, a lossless compressor for floating-point streams
#30It splits the input into adaptively-sized blocks (quanta), runs a competition between many specialized codecs on each block, and emits the smallest result. This is, for lack of a better term, a "metacompressor", but it will be interesting to see which of the choices end up dominating; in my past experiences with metacompression, one algorithm is usually consistently ahead.
I’ve never heard of a metacompressor before, what others exist?
Even back when star wars episode 1 came out with quicktime files of the trailer there were multiple codecs used in the same video file to make it look as good as possible.
Making up a new term isn't necessary, this has been done and everyone just called it compression.