11–20 of 24 posts
Congrats ! If anyone is interested in the clearlinux optimized zstd build config ( imho it is useful for the 1.5.5 building ) https://github.com/clearlinux-pkgs/zstd/blob/main/zstd.spec ( CFLAGS, 4 patch )
What's stopping those 4 patches from being upstreamed? What advantages or disadvantages am I getting from them?
Earlier quoted context omitted.
What's stopping those 4 patches from being upstreamed? What advantages or disadvantages am I getting from them?
Looking at it it seems to require avx2. And use the "m" mode for fopen(), which I have no idea what it does, it's not in any documentation I can find.
m (since glibc 2.3)
Attempt to access the file using mmap(2), rather than I/O system calls (read(2), write(2)). Currently, use of mmap(2) is attempted only for a file opened for reading.
Congrats ! If anyone is interested in the clearlinux optimized zstd build config ( imho it is useful for the 1.5.5 building ) https://github.com/clearlinux-pkgs/zstd/blob/main/zstd.spec ( CFLAGS, 4 patch )
What's stopping those 4 patches from being upstreamed? What advantages or disadvantages am I getting from them?
* Change default value for some parameters, that can't be upstream due to compatibility reasons.
* Change build flag / macro for optimization and/or workaround compiler bugs.
Earlier quoted context omitted.
Looking at it it seems to require avx2. And use the "m" mode for fopen(), which I have no idea what it does, it's not in any documentation I can find.
From my fopen(2) docs: m (since glibc 2.3) Attempt to access the file using mmap(2), rather than I/O system calls (read(2), write(2)). Currently, use of mmap(2) is attempted only for a file opened for reading.
> corruptions are detected thanks to the checksum That's good to know because, for important things, I test the archive before throwing the original data away. Not to single out zstd but it's a good opportunity to be reminded: if you make backups, test your backups! A bug like this can also be introduced, not only fixed. I'm not saying it's likely, or that you're likely to be affected by it, but for things you care a…
To begin with, it requires the distance between 2 consecutive matches to be exactly 65536. This is extremely rare. Mountains of files never generate such a situation. Then it needs to employ a repcode match as the match following the 65536 literals. Repcode matches are more common after short literal lengths (1-3 bytes). Needless to say, 65536 is far from this territory, so it's uncommon to say the least. Finally, the block splitter must be active (hence only high compression modes), and it must decide to split the block exactly at the boundary between the literals and the repcode match.
So this is not 0, since Google found a sample, but all these conditions _together_ have an astronomically low chance to happen, as in competitive with winning the Powerball jackpot. I wouldn't worry so much for my own dozens of archives.
Earlier quoted context omitted.
From my fopen(2) docs: m (since glibc 2.3) Attempt to access the file using mmap(2), rather than I/O system calls (read(2), write(2)). Currently, use of mmap(2) is attempted only for a file opened for reading.
I'm very skeptical that this actually makes things faster. For one, zstd is definitely going to be CPU bound rather than I/O bound. But even if it wasn't, there shouldn't be a performance benefit to reading a file sequentially using mmap compared to reading it sequentially using read, which you can easily verify yourself by writing a small program that just reads files both ways.
Earlier quoted context omitted.
> That's good to know because, for important things, I test the archive before throwing the original data away. Compression corruptions are worse then regular corruption due to the cascading impact. A corrupt sector can be replaced inline but a corrupt compressed file will generally destroy everything downstream from the error. Big +1 to actually verifying the round trip. Backups that aren’t tested through an actual…
Would pigz's parallel compression prevent that failure case, at least limiting it to its block size of a default 128K? https://github.com/madler/pigz/blob/master/pigz.1#L43-L45 That would be a nice extra benefit, besides the speedup from being multithreaded. (I assume zstd also does multithreading but for those stuck with gzip, this is a drop-in replacement.) Edit: bzip2 apparently does the same, "bzip2 compresses fi…
Other gzip variant formats like bgzip also make the chunks compressed in parallel completely independent. This results in ~3% worse compression ratio depending on the use case.
Note that another problem with bit flips and other errors in compression formats is that most decompression tool will simply quit on the first error even if the rest of the data could still be recovered.
Yes, bz2 is also more robust against errors because of the independent blocks.
Anybody willing to explain what "nb" stands for in this context?
> due to the nb and complexity of simultaneous conditions Anybody willing to explain what "nb" stands for in this context?
> due to the nb and complexity of simultaneous conditions Anybody willing to explain what "nb" stands for in this context?