Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of ZFS on Linux

clusterhq.com

Post reply on HN
Print view View original

121–125 of 125 posts

Re: The State of ZFS on Linux

#121

Post by 1amzave »

nailer wrote: post #37

Putting production data on a driver maintained outside the mainline Linux kernel is a bad idea. That isn't a licensing argument - I'm happy to use a proprietary nvidia.ko for gaming tasks, for example, because I won't be screwing up anyone's data if it breaks.

Maybe the odds of it doing so are smaller, but if you think a broken nvidia.ko can't screw up anyone's data I think you're simply mistaken.

Say Nvidia's driver has a use-after-free bug: it kmalloc()s a buffer, kfree()s it, then a filesystem kmalloc()s something and gets allocated the same buffer. If nvidia.ko then decides it still wants to use that buffer and writes something into it...kablooie.

Unless you start running some microkernel-ish thing with drivers each running in their own distinct address spaces, you're going to have a hard time avoiding this possibility.

Re: The State of ZFS on Linux

#122

Post by db48x »

ansible wrote: post #111
ryao wrote: post #106

Earlier quoted context omitted.

Saying that a checksum is weak in a cryptographic sense would mean that it is possible for a malicious intelligence to generate collisions. This is not what I meant when I said that CRC was weak. Instead, I meant that it is trivial for ordinary glitches to corrupt data in ways that the checksums fail to detect. Things byte swaps and non-adjacent double bit flips are all that is necessary. The link I provided elaborat…

This is not what I meant when I said that CRC was weak. Instead, I meant that it is trivial for ordinary glitches to corrupt data in ways that the checksums fail to detect. Things byte swaps and non-adjacent double bit flips are all that is necessary. The link I provided elaborates on this ... Since we're talking about data corruption that has managed to not be detected by the hard drive's own firmware (a read error…

It's pretty scary; data on ordinary filesystems just evaporates over time. Undetected bit-flips every few terabytes from the drive itself, flaky cables and controllers, bit flips in your non-ecc ram, ghost writes, misaligned reads, firmware bugs, cheap port multipliers, etc, etc.

I've had a zfs filesystem for a few years now and twice it's detected and corrected an error that would have been silent data corruption in a lesser filesystem.

Re: The State of ZFS on Linux

#123

Post by Andys »

ryao wrote: post #119
Andys wrote: post #74

I used ZFSonLinux on my laptop and workstation for a couple of years now, with Ubuntu, without any major problems. When I tried to use it in production, I didn't get data loss but I hit problems: * Upgrading is a crapshoot: Twice, it failed to remount the pool after rebooting, and needed manual intervention. * Complete pool lockup: in an earlier version, the pool hung and I had to reboot to get access to it again. If…

I mentioned most of these issues in the supplementary blog posts. Here is where each stands: * There are issues when upgrading because the initramfs can store an old copy of the kernel module and the /dev/zfs interface is not stabilized. This will be addressed in the next 6 months by a combination of two things. The first is /dev/zfs stabilization. The second is bootloader support for dynamic generation of initramfs…

Great reply :)

Your last two points are a little weak. Ultimately ZFS does more stuff, so on an SSD its going to be slower than a filesystem which doesn't have all these extra features if you aren't using them. I think there's a trade-off to be had.

All software has bugs, especially ZFS. You could argue it is easier to developer, test, and maintain individual device-mapper building blocks.

Re: The State of ZFS on Linux

#124

Post by colin_mccabe »

db48x wrote: post #122
ansible wrote: post #111

Earlier quoted context omitted.

This is not what I meant when I said that CRC was weak. Instead, I meant that it is trivial for ordinary glitches to corrupt data in ways that the checksums fail to detect. Things byte swaps and non-adjacent double bit flips are all that is necessary. The link I provided elaborates on this ... Since we're talking about data corruption that has managed to not be detected by the hard drive's own firmware (a read error…

It's pretty scary; data on ordinary filesystems just evaporates over time. Undetected bit-flips every few terabytes from the drive itself, flaky cables and controllers, bit flips in your non-ecc ram, ghost writes, misaligned reads, firmware bugs, cheap port multipliers, etc, etc. I've had a zfs filesystem for a few years now and twice it's detected and corrected an error that would have been silent data corruption in…

Serious servers use ECC RAM so bit-flips in memory are not an issue. CRC32 is also pretty robust against single-bit errors. If you think about it, you can't really make any guarantees for non-ECC systems. Flip the wrong bit, and the kernel will delete all your data. There is an if statement somewhere waiting to make this happen, and if the program code can randomly change, eventually it will.

Typical disk corruption patterns are a whole sector of zeroes being read, or reading data that belonged somewhere else. If you want to prove that ZFS's checksum is better, you need to prove that it's better against the patterns of corruption that actually occur.

Re: The State of ZFS on Linux

#125

Post by colin_mccabe »

ryao wrote: post #106
colin_mccabe wrote: post #101

Earlier quoted context omitted.

As others have commented, CRC32 or CRC64 being "weak" in a cryptographic sense doesn't mean it's not suitable for detecting disk errors. x86 has a crc32 instruction, making it far faster than any other option. By the way, TCP doesn't use CRC32... it uses a simple additive checksum. Chris Mason wrote most of btrfs while he was at Oracle. Hence, the GPLv2 grants an implicit patent license to whatever patents Oracle mig…

Saying that a checksum is weak in a cryptographic sense would mean that it is possible for a malicious intelligence to generate collisions. This is not what I meant when I said that CRC was weak. Instead, I meant that it is trivial for ordinary glitches to corrupt data in ways that the checksums fail to detect. Things byte swaps and non-adjacent double bit flips are all that is necessary. The link I provided elaborat…

As for patent grants, you are thinking of the GPLv3. The GPLv2 does not provide any patent grant.

This is incorrect. See http://en.swpat.org/wiki/GPLv2_and_patents

Post reply on HN