Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In OpenZFS and Btrfs, everyone was just guessing

phoronix.com

Post reply on HN
Print view View original

1–10 of 172 posts

Re: In OpenZFS and Btrfs, everyone was just guessing

#3

Post by nabla9 »

> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS.

When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening.

In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once a month or so. Oracle had to send on-site team to monitor the database operations to catch the bug in action. They basically had their own datacenter to record all the logs from database, networks and operating system.

It took weeks for them to catch the bug. It was related to certain network driver in the operating system and the bug manifested only with some specific network traffic pattern.

Re: In OpenZFS and Btrfs, everyone was just guessing

#7

Post by lvncelot »

szpght wrote: post #5

The bug was finally fixed with the following commit: https://github.com/openzfs/zfs/pull/15579/commits/679738cc40... It looks like they managed to exactly pin down what was happening.

Sidenote: What a great commit message. The expanded comment explains why both checks are necessary, but the commit message gives so much more context for anyone wondering and `git blame`ing that line.

Re: In OpenZFS and Btrfs, everyone was just guessing

#8

Post by rini17 »

lmz wrote: post #6

Reading the comment re: what the FS dev teams should do by someone who I bet didn't pay a dime for the software just upsets me.

Mr. Behlendorf seems to work at an nonprofit that gets grants from DARPA. So if they are US citizen, they did in fact pay for this.

Re: In OpenZFS and Btrfs, everyone was just guessing

#9

Post by kalleboo »

lmz wrote: post #6

Reading the comment re: what the FS dev teams should do by someone who I bet didn't pay a dime for the software just upsets me.

Also going on about how everything was done right "back in my day". Which doesn't sound true to me, every old graybeard story I read was full of complaining about how buggy and unusable the systems by the old tech giants were.

Re: In OpenZFS and Btrfs, everyone was just guessing

#10

Post by xoa »

First, there is a FreeBSD Errata Notice for this that offers an nice quick collection of the various bugs and subsequent repairs, with links to summaries, for anyone who is catching up on this issue:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275308

Second, I don't like the editorialization of this title ("In OpenZFS and Btrfs, everyone was just guessing") at all. No, nobody was "just guessing", but as far as I know there is no featureful FS that has undergone formal verification. It's a large codebase started by a now long defunct company that solved critical problems and delivered a lot of value, but there absolutely can be issues lurking over long time periods. It's a testament if anything that work and usage did uncover the issue, it wasn't brushed away at all but instead drilled down on and solved in short order, and now the already extensive test suites are expanded again in an organized way. And spekaing of "critical problems":

Third, a lot of the commentary around these sorts of things seems to indulge in noticing the rare misses while ignoring many hits. Amongst other reasons part of the core motivation for me to switch to ZFS 100% in 2010/2011 or so and doggedly stick with it ever since was precisely because I experienced data rot (permanent corruption) with my data under previous filesystems where that WAS NOT A BUG. HFS/UFS/NTFS/XFS/whatever, none of them offer any guarantees of data integrity in the first place! Bit flips somewhere, hardware has issues, copying has noise, whatever? RAID-5 write hole? Those or lots of other things are not a bugs at all in old filesystems, because that's just how those primitive things were. I've been carrying forward data since my first Apple IIe, and went back to find some of my old work, early digital photos and drawings I cared about, and somewhere along the line it had gotten mucked up. I know not where or when, because there was no chain of checksumming and trust that would have a chance to alert me. It's impossible for a human to keep up with terabytes of data manually, it has to be fully automated, baked in. At least in ZFS there being any corruption is a drop-everything big deal problem that is incredibly rare and niche and serious people will care very much about. Pretending the old stuff was good or even acceptable is pure bullshit.

It doesn't have to be mathematically perfect to deliver value, and importantly to be far superior to everything that came before. And it's inarguable that it's been battle tested very very hard for a very long time at this point. It's certainly saved my bacon a few times.

Post reply on HN