Fc, a lossless compressor for floating-point streams
1–10 of 35 posts
Re: Fc, a lossless compressor for floating-point streams
#2I built "fc", a C library for compressing streams of 64-bit floating-point values without quantization.
It is not trying to replace zstd or lz4. The idea is narrower: take blocks of doubles, try a set of float-specific predictors/transforms/coders, and emit whichever representation is smallest for that block.
It is aimed at time-series, scientific, simulation, and analytics data where the numbers often have structure: smooth curves, repeated values, fixed increments, periodic signals, predictable deltas, or low-entropy mantissas.
The API is intentionally small: "fc_enc", "fc_dec", a config struct, and a few counters to inspect which modes won. Decode is parallel and meant to be fast; encode spends more CPU searching for a better representation.
Current caveats: x86-64 only for now, tuned for IEEE-754 doubles, research-grade rather than production-hardened.
Re: Fc, a lossless compressor for floating-point streams
#3I built "fc", a C library for compressing streams of 64-bit floating-point values without quantization. It is not trying to replace zstd or lz4. The idea is narrower: take blocks of doubles, try a set of float-specific predictors/transforms/coders, and emit whichever representation is smallest for that block. It is aimed at time-series, scientific, simulation, and analytics data where the numbers often have structure…
Does it assume the floats come from photos or sound or something?
Re: Fc, a lossless compressor for floating-point streams
#4I built "fc", a C library for compressing streams of 64-bit floating-point values without quantization. It is not trying to replace zstd or lz4. The idea is narrower: take blocks of doubles, try a set of float-specific predictors/transforms/coders, and emit whichever representation is smallest for that block. It is aimed at time-series, scientific, simulation, and analytics data where the numbers often have structure…
Does it assume the floats come from photos or sound or something?
It is intended t obe mainly source agnostic (will try to add custom source predictors too). The idea is to treat input as an ordered stream of doubles and look for numeric structure like repeats, smooth deltas, fixed increments, or low-entropy bits. Target presentlyis scientific/time-series/simulation/analytics data, not photos or sound.
Re: Fc, a lossless compressor for floating-point streams
#5I see you have ALP, but have you tried Chimp128 or Arrow's byte stream split?
Re: Fc, a lossless compressor for floating-point streams
#6Another library in this space is pcodec; I'd appreciate a comparison of the two.
Re: Fc, a lossless compressor for floating-point streams
#7The question is, how close can OpenLZ come? (This is from the same people who develop zstd, but suitable for structured data in a generic way.)
Re: Fc, a lossless compressor for floating-point streams
#8I built "fc", a C library for compressing streams of 64-bit floating-point values without quantization. It is not trying to replace zstd or lz4. The idea is narrower: take blocks of doubles, try a set of float-specific predictors/transforms/coders, and emit whichever representation is smallest for that block. It is aimed at time-series, scientific, simulation, and analytics data where the numbers often have structure…
What do you mean by decode is parallel?
Re: Fc, a lossless compressor for floating-point streams
#9Earlier quoted context omitted.
Does it assume the floats come from photos or sound or something?
It is intended t obe mainly source agnostic (will try to add custom source predictors too). The idea is to treat input as an ordered stream of doubles and look for numeric structure like repeats, smooth deltas, fixed increments, or low-entropy bits. Target presentlyis scientific/time-series/simulation/analytics data, not photos or sound.
isn't sound a time series? I guess it's not usually 64-bit doubles.
Re: Fc, a lossless compressor for floating-point streams
#10It splits the input into adaptively-sized blocks (quanta), runs a competition between many specialized codecs on each block, and emits the smallest result.
This is, for lack of a better term, a "metacompressor", but it will be interesting to see which of the choices end up dominating; in my past experiences with metacompression, one algorithm is usually consistently ahead.