Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZSTD 1.5.5 is released with a corruption fix found at Google

github.com

Post reply on HN
Print view View original

21–24 of 24 posts

Re: ZSTD 1.5.5 is released with a corruption fix found at Google

#21

Post by RaisingSpear »

lucb1e wrote: post #7
koolba wrote: post #6

Earlier quoted context omitted.

> That's good to know because, for important things, I test the archive before throwing the original data away. Compression corruptions are worse then regular corruption due to the cascading impact. A corrupt sector can be replaced inline but a corrupt compressed file will generally destroy everything downstream from the error. Big +1 to actually verifying the round trip. Backups that aren’t tested through an actual…

Would pigz's parallel compression prevent that failure case, at least limiting it to its block size of a default 128K? https://github.com/madler/pigz/blob/master/pigz.1#L43-L45 That would be a nice extra benefit, besides the speedup from being multithreaded. (I assume zstd also does multithreading but for those stuck with gzip, this is a drop-in replacement.) Edit: bzip2 apparently does the same, "bzip2 compresses fi…

One thing to note is that DEFLATE (underlying algorithm that Gzip uses) doesn't indicate the length of blocks, or make it easy to figure out where they start/end without decoding everything proceeding it. This is likely why pigz can only parallelize compression, but not decompression.

But even if you could identify the block boundaries up front, DEFLATE doesn't reset the LZ77 window on a new block, so corruption could still seep through to the end.

Re: ZSTD 1.5.5 is released with a corruption fix found at Google

#23

Post by azatom »

azatom wrote: post #22

I don't know if it's related, but one time I had corruption too. I was playing with large sparse (ntfs) files and used zstd for temporarily store those. It was probably low compression settings and I couldn't reproduce, so I'd thought I messed up something.

oh, I remembered, it was not all zstd fault. Somehow because zstad read not only the allocated space but the whole fole, the zeros too, windows tried to allocate the whole file in memory, which was not enough, and zstd did not give (outofmemory) eror message. (It was not ntfs compressed, only sparse!) Later (maybe windowsupdate happend) tried to reproduce, but worked with larger file with memory.

I don't know zstd how could handle the outofmemory, but the sparse reading might not be perfect.

Re: ZSTD 1.5.5 is released with a corruption fix found at Google

#24

Post by marssaxman »

akx wrote: post #20
marssaxman wrote: post #18

> due to the nb and complexity of simultaneous conditions Anybody willing to explain what "nb" stands for in this context?

Number. Common abbreviation around Francophone programmers (see https://fr.wikipedia.org/wiki/NB : "L'abréviation Nb peut faire référence à : nombre")

Thank you! Never would have guessed that on my own.
Post reply on HN