Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Silent Data Corruption Is Real

changelog.complete.org

Post reply on HN
Print view View original

101–110 of 154 posts

Re: Silent Data Corruption Is Real

#101

Post by ReidZB »

bascule wrote: post #93
ak217 wrote: post #66

Earlier quoted context omitted.

crc32c is not weakish, and was chosen for a reason: crc32c has widespread hardware acceleration support that remains faster than any hash, and crc32c can be computed in parallel (unlike a hash, it has no hidden state, so you can sum independently computed block checksums to get the overall blob checksum). Bitrot detection doesn't need a cryptographic hash. You may want a hash for other purposes (like if you somehow t…

Bitrot detection doesn't need a cryptographic hash. Not only is a cryptographic hash unnecessary, under certain circumstances it will actually do a worse job. Cryptographic hashes operate under a different set of constraints than error detecting code. With an error detecting code, it's desirable to guarantee a different checksum in the event of a bitflip. With a cryptographic random oracle, this is not the case: we w…

In practice, the chance of a digest collision between two messages that differ in a single bit is exceedingly small for any secure cryptographic hash function. It's so small that it's practically not worth considering. Cryptographers are incredibly careful in building and ensuring proper diffusion in cryptographic hash functions.

Re: Silent Data Corruption Is Real

#102

Post by mrb »

bascule wrote: post #93
ak217 wrote: post #66

Earlier quoted context omitted.

crc32c is not weakish, and was chosen for a reason: crc32c has widespread hardware acceleration support that remains faster than any hash, and crc32c can be computed in parallel (unlike a hash, it has no hidden state, so you can sum independently computed block checksums to get the overall blob checksum). Bitrot detection doesn't need a cryptographic hash. You may want a hash for other purposes (like if you somehow t…

Bitrot detection doesn't need a cryptographic hash. Not only is a cryptographic hash unnecessary, under certain circumstances it will actually do a worse job. Cryptographic hashes operate under a different set of constraints than error detecting code. With an error detecting code, it's desirable to guarantee a different checksum in the event of a bitflip. With a cryptographic random oracle, this is not the case: we w…

"Not only is a cryptographic hash unnecessary, under certain circumstances it will actually do a worse job."

A cryptographic hash is unnecessary, but as you point out in the next to last paragraph, it is statistically improbable that it will do a worse job. Because collisions are statistically improbable.

Re: Silent Data Corruption Is Real

#103

Post by duskwuff »

dandelion_lover wrote: post #61
danjoc wrote: post #8

Closed source firmware on drives contain bugs that corrupt data. Are there any drives, available anywhere, that have open source firmware?

There is a corresponding project: http://www.openssd-project.org

Note that it's essentially a dead project. The "NEW!!!" platform mentioned on their homepage is from 2014.

Re: Silent Data Corruption Is Real

#104

Post by newman314 »

tnorgaard wrote: post #68
tombrossman wrote: post #27

Earlier quoted context omitted.

Any idea if ZFS plays well with Ubuntu's full-disk encryption? I've used the FDE option at install for years and every time I upgrade (I wipe & reinstall every year or so) I try to understand how to first set up ZFS, then FDE, and then I realize it's far too complicated for me. Any good tutorials or setup guides that even a moron could understand? I've got a pretty good setup now with a fairly complex fstab, multiple…

We run with ZFS over LUKS encrypted volumes in production on AWS ephemeral disks and have done so in over two years on Ubuntu 14.04 and 16.04. The major issue for us has been getting the startup order right, as timing issues does occur once you have many instances. To solve this, we use upstart (14.04) and systemd (16.04) together with Puppet to control the ordering. Performance wise it does fairly well, our benchmar…

What is the right order?

Re: Silent Data Corruption Is Real

#105

Post by johnramsden »

dom0 wrote: post #23
elfchief wrote: post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

btrfs only uses CRC32c which is weakish. ZFS is great but not exactly portable. I started to use Borg now for archiving purposes as well, not just backup. For me (low access concurrency, i.e. single or at most "a few" users) that works very well. Portable + strong checksumming + strong crypto + mountable + reasonable speed (with prospect of more) is a good package. It doesn't solve error correction, though.

ZFS is probably one of your better options if you want portability. It works on most Unix like operating systems, and due to its zfs send/recv capability you can send your data all over the place - and can trust that it will actually end up the same on the other end. If that's not portability I don't know what is.

If what you're saying is it's not portable to Windows then sure, but compared to most filesystems it's extremely portable.

Re: Silent Data Corruption Is Real

#106

Post by stubish »

elfchief wrote: post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

zfs is supported in Ubuntu by Canonical, both in the 'out of the box and recommended' sense and the pay-for-support sense via Ubuntu Advantage.

Unfortunately, you still can't get a root zfs partition without hassles.

Re: Silent Data Corruption Is Real

#107

Post by dom0 »

seiferteric wrote: post #15
nayuki wrote: post #6

Earlier quoted context omitted.

Btrfs has experienced some data loss bugs in recent memory. It looks like ZFS is the only remaining option. https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Da... https://www.spinics.net/lists/linux-btrfs/msg59190.html https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg...

Yes I have tried to use btrfs several times for work projects and personally because I was very excited about it but every time I have ran into severe bugs even though it was said to be "stable". I have given up for now, maybe I will check back in a couple more years.

Funny, my btrfs story is somewhat similar. I don't hit severe bugs, though, just severe performance issues. E.g. btrfs on / always sooner or later means for me that the entire IO to the disk that's on will be _thrashed_ during system updates or package installs / uninstalls.

Re: Silent Data Corruption Is Real

#108

Post by XorNot »

drvdevd wrote: post #52
tombrossman wrote: post #27

Earlier quoted context omitted.

Any idea if ZFS plays well with Ubuntu's full-disk encryption? I've used the FDE option at install for years and every time I upgrade (I wipe & reinstall every year or so) I try to understand how to first set up ZFS, then FDE, and then I realize it's far too complicated for me. Any good tutorials or setup guides that even a moron could understand? I've got a pretty good setup now with a fairly complex fstab, multiple…

I use it all over with LUKS on Ubuntu. It works fine, but there's one little hitch: when calling anything that ultimately calls grub-probe (e.g. apt-get upgrade), you have to symlink the decrypted device mapper volume up a layer into /dev because grub-probe can't seem to find the ZFS vdev(s) otherwise. ie: "ln -s /dev/mapper/encrypted-zfs-vdev /dev". This is in fact the case on every Linux distribution I've run ZFS o…

The other problem I've found is that grub's update-grub scripts do not handle mirrored ZFS volumes well at all - they wind up just spraying doubled up invalid commands everywhere even up to yakkety so far.

I've had it on my backlog to at some point go in and sort out my initramfs's insanity when it comes to handling crypt'd disks in general - it should be a lot less brittle then it is.

Re: Silent Data Corruption Is Real

#109

Post by weitzj »

newman314 wrote: post #104
tnorgaard wrote: post #68

Earlier quoted context omitted.

We run with ZFS over LUKS encrypted volumes in production on AWS ephemeral disks and have done so in over two years on Ubuntu 14.04 and 16.04. The major issue for us has been getting the startup order right, as timing issues does occur once you have many instances. To solve this, we use upstart (14.04) and systemd (16.04) together with Puppet to control the ordering. Performance wise it does fairly well, our benchmar…

What is the right order?

Since ZFS will run on blocklevel devices and you want to get the ZFS benefits of Snapshots/compression/(deduplication), in my opinion it makes sense to do the encryption at the blocklevel, i.e. LUKS has to provide decrypted block level devices before ZFS searches for its zpools. When ZFS native encryption is available on Linux this will be different, since you much finer control on what to encrypt and you can keep all ZFS features.

So:

First decrypt LUKS (we are doing this in GRUB) Then mount zpool(s)

Re: Silent Data Corruption Is Real

#110

Post by weitzj »

XorNot wrote: post #108
drvdevd wrote: post #52

Earlier quoted context omitted.

I use it all over with LUKS on Ubuntu. It works fine, but there's one little hitch: when calling anything that ultimately calls grub-probe (e.g. apt-get upgrade), you have to symlink the decrypted device mapper volume up a layer into /dev because grub-probe can't seem to find the ZFS vdev(s) otherwise. ie: "ln -s /dev/mapper/encrypted-zfs-vdev /dev". This is in fact the case on every Linux distribution I've run ZFS o…

The other problem I've found is that grub's update-grub scripts do not handle mirrored ZFS volumes well at all - they wind up just spraying doubled up invalid commands everywhere even up to yakkety so far. I've had it on my backlog to at some point go in and sort out my initramfs's insanity when it comes to handling crypt'd disks in general - it should be a lot less brittle then it is.

We are rolling our own update grub as a bash script, which does some sanity checks:

- is the crypto module still present in GRUB config?

- is grub running in text only mode? Otherwise we cannot see the LUKS prompt to decrypt the devices.

- does initramfs know about all mirror devices (instead of an early return after the first)

Post reply on HN