Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In OpenZFS and Btrfs, everyone was just guessing

phoronix.com

Post reply on HN
Print view View original

21–30 of 172 posts

Re: In OpenZFS and Btrfs, everyone was just guessing

#21

Post by jsiepkes »

XorNot wrote: post #13
xoa wrote: post #10

First, there is a FreeBSD Errata Notice for this that offers an nice quick collection of the various bugs and subsequent repairs, with links to summaries, for anyone who is catching up on this issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275308 Second, I don't like the editorialization of this title ("In OpenZFS and Btrfs, everyone was just guessing") at all. No, nobody was "just guessing", but as far as…

Yep, this is how I ended up running ZFS. Because with XFS the answer to "oh no a power cut" was "lol, a bunch of files might be zeros. Figure out which ones." The whole weird implication that there's something else which is perfect is always bizarre: there isn't, everyone knows that, it's extremely non-trivial to do that.

> Because with XFS the answer to "oh no a power cut" was "lol, a bunch of files might be zeros. Figure out which ones."

Got bitten by that one once as well. I recall the XFS FAQ had something like in it like: "If you know it hurts, then don't do it."

Re: In OpenZFS and Btrfs, everyone was just guessing

#22

Post by donmcronald »

reacharavindh wrote: post #19

I’m not going to corroborate the entire article, but my experience from an outage 2 days ago on our new ZFS on Linux file server has left a bad taste despite years of great use with ZFS in the FreeBSD world. Ok. So, We hit the deadlock that was fixed here:-> https://github.com/rohan-puri/zfs/commit/8e4d086c13c16bc461b... but it never got merged to the master or release as far as I see it Current Master: https://githu…

Isn't it there?

https://github.com/openzfs/zfs/commit/fd7265c646f40e364396af...

    git branch --contains fd7265c
    
    * master
Since 0.8.

    git branch -r --contains fd7265c | head
    
      origin/HEAD -> origin/master
      origin/compat-5.15-META
      origin/issue-14573-backport
      origin/master
      origin/revert-14721-remove-quota-zap
      origin/zfs-0.8-release
      origin/zfs-0.8.7-staging
      origin/zfs-2.0-release
      origin/zfs-2.0.3-staging
      origin/zfs-2.0.4-staging

Re: In OpenZFS and Btrfs, everyone was just guessing

#23

Post by [deleted] »

dmpk2k wrote: post #16

While I am firmly in the ZFS camp, my feeling is that there has indeed been a gradual slide in the disciplined development of ZFS. After some initial bumps in brand new software, the Sun kernel group did a good job of avoiding corruption. Once Sun was absorbed by Oracle, OSS ZFS moved into illumos, who were overall quite good at doing the same (although they had less resources to play with). OpenZFS brought ZFS to th…

[deleted]

Re: In OpenZFS and Btrfs, everyone was just guessing

#24

Post by seanw444 »

mcpackieh wrote: post #12

I have been more than satisfied with single-drive Btrfs with OpenSUSE so far, but nevertheless I can't wait for bcachefs to prove itself. On paper at least, bcachefs seems like the most attractive of the three.

ZFS will still be preferable for production for a long while until bcachefs has proven itself, and bug reports slow down. That said, I'm also stoked for bcachefs.

ZFS has always appeared to me like a bloated mess. Btrfs is more minimal, but too much so, in that it doesn't do parity (at least very reliably), which sucks because I don't always want mirroring. And it has a reputation for gobbling your data up.

If bcachefs can provide all the most useful features of ZFS, while being minimal and baked into Linux by default, man that will be amazing.

Re: In OpenZFS and Btrfs, everyone was just guessing

#25

Post by onetimeuse92304 »

nabla9 wrote: post #3

> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once…

The hardest bug I have ever worked with was an embedded device losing all data on the flash chip.

But the issue was, this was happening even after we removed all instructions to delete data from the flash.

The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness when the flash was connected to controller outside the board.

The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died on average. But we had millions of them in the wild and we would get hundreds or thousands die on our customers every week.

It took half a year of debugging and tens of millions of dollars in losses.

We found that one of unrelated components on the board emitted too much noises. And a software change to speed up writes to the flash removed a special sequence of that preceded and followed every one word write to the flash. This special sequence protected the flash from accepting a random noise as a command. Usually, the command would be something harmful or meaningless. But sometimes it would be interpreted as a real command -- and one of these was a command to clear entire flash chip.

Re: In OpenZFS and Btrfs, everyone was just guessing

#26

Post by Fire-Dragon-DoL »

onetimeuse92304 wrote: post #25
nabla9 wrote: post #3

> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once…

The hardest bug I have ever worked with was an embedded device losing all data on the flash chip. But the issue was, this was happening even after we removed all instructions to delete data from the flash. The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness…

Oh god, that's insane

Re: In OpenZFS and Btrfs, everyone was just guessing

#27

Post by solarkraft »

lvncelot wrote: post #7
szpght wrote: post #5

The bug was finally fixed with the following commit: https://github.com/openzfs/zfs/pull/15579/commits/679738cc40... It looks like they managed to exactly pin down what was happening.

Sidenote: What a great commit message. The expanded comment explains why both checks are necessary, but the commit message gives so much more context for anyone wondering and `git blame`ing that line.

Totally. Also: I've never seen Sponsored-by:

> Signed-off-by: Rob Norris

> Sponsored-by: Klara, Inc.

> Sponsored-by: Wasabi Technology, Inc.

Re: In OpenZFS and Btrfs, everyone was just guessing

#29

Post by solarkraft »

donmcronald wrote: post #18

I don't know about BTRFS, but with ZFS there's a lot of bad information and assumptions floating around. It's one of my favorite topics to ask AI about because it shows the obvious deficiencies in AI as it regurgitates all the bad info. So when you go to the issue tracker, it seems plausible to me that you're going to find issues where the devs can't explain what's happening because the user(s) might be doing somethi…

A lot of end users run ZFS through TrueNAS/FreeNAS and I indeed found that community to spread a lot of misinfo and un-actionable advice. I liken them to gamers lapping up any snake oil solution to get 0.1 fps more. I would suggest anyone trying to learn about ZFS to avoid it, it definitely was counter-productive to me.

Reddit, random blogs and official documentation (often that by Oracle) turned out to be much better resources.

Re: In OpenZFS and Btrfs, everyone was just guessing

#30

Post by ceeam »

dmpk2k wrote: post #16

While I am firmly in the ZFS camp, my feeling is that there has indeed been a gradual slide in the disciplined development of ZFS. After some initial bumps in brand new software, the Sun kernel group did a good job of avoiding corruption. Once Sun was absorbed by Oracle, OSS ZFS moved into illumos, who were overall quite good at doing the same (although they had less resources to play with). OpenZFS brought ZFS to th…

Too much source, eventually a human cannot follow it. A common software (and not only) problem.

Maybe sometimes once in 2-3 decades it's worth taking the lessons learned and re-doing the thing? Simplifying and streamlining it greatly in the process?

Like, this bug, the file holes - do we even need them now, with very good compression for once?

Post reply on HN