Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In OpenZFS and Btrfs, everyone was just guessing

phoronix.com

Post reply on HN
Print view View original

41–50 of 172 posts

Re: In OpenZFS and Btrfs, everyone was just guessing

#41

Post by bsder »

rollcat wrote: post #20
zogomoox wrote: post #11

Wouldn't a file system in a slightly higher level language solve some of this pain? (edit: in terms of being able to understand the system)

You're at odds with the desire (necessity, actually) to have precise low-level control, not just of the in-memory layout of the data structures, but also of the performance characteristics of your code (e.g. no unnecessary pointer-chasing). Higher-level languages tend to make it easier to pile up abstractions; we want the orthogonal property of making it harder to shoot yourself in the foot. You can look at ZFS for i…

I like Rust, but you are going to be writing a lot of unsafe Rust for a filesystem implementation. Multiple processes are writing to the filesystem simultaneously. "Ownership" is fuzzy and is moving around.

At that point, is Rust buying you anything for how much it's going to get in your way?

I really don't see an advantage to Rust when operating at these kinds of low levels.

Re: In OpenZFS and Btrfs, everyone was just guessing

#42

Post by vardump »

nabla9 wrote: post #3

> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once…

Network drivers contribute to "bugs" surprisingly often. I've had my share of issues with them.

I'd go as far to claim this: If you're shipping a product that relies on networking at your customer's premises you basically need to maintain a list of cards and drivers that you support. Especially if you're doing anything weird, where "weird" means anything but bog standard TCP.

Re: In OpenZFS and Btrfs, everyone was just guessing

#43

Post by vardump »

onetimeuse92304 wrote: post #25
nabla9 wrote: post #3

> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once…

The hardest bug I have ever worked with was an embedded device losing all data on the flash chip. But the issue was, this was happening even after we removed all instructions to delete data from the flash. The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness…

Another source of bogus commands can be brownouts. When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring.

Re: In OpenZFS and Btrfs, everyone was just guessing

#44

Post by solarkraft »

emaste wrote: post #35
solarkraft wrote: post #27

Earlier quoted context omitted.

Totally. Also: I've never seen Sponsored-by: > Signed-off-by: Rob Norris > Sponsored-by: Klara, Inc. > Sponsored-by: Wasabi Technology, Inc.

This is a longstanding tradition from FreeBSD. A list of our commit message trailers: https://docs.freebsd.org/en/articles/committers-guide/#_incl... . "Sponsored by:" search in FreeBSD commit messages: https://freshbsd.org/?q=%22Sponsored+by%3A%22

I like it! Seems like a great way to incentivise companies to contribute.

Re: In OpenZFS and Btrfs, everyone was just guessing

#45

Post by frankjr »

szpght wrote: post #5

The bug was finally fixed with the following commit: https://github.com/openzfs/zfs/pull/15579/commits/679738cc40... It looks like they managed to exactly pin down what was happening.

A more "correct" fix has been posted https://github.com/openzfs/zfs/pull/15615

Re: In OpenZFS and Btrfs, everyone was just guessing

#46

Post by steveklabnik »

bsder wrote: post #41
rollcat wrote: post #20

Earlier quoted context omitted.

You're at odds with the desire (necessity, actually) to have precise low-level control, not just of the in-memory layout of the data structures, but also of the performance characteristics of your code (e.g. no unnecessary pointer-chasing). Higher-level languages tend to make it easier to pile up abstractions; we want the orthogonal property of making it harder to shoot yourself in the foot. You can look at ZFS for i…

I like Rust, but you are going to be writing a lot of unsafe Rust for a filesystem implementation. Multiple processes are writing to the filesystem simultaneously. "Ownership" is fuzzy and is moving around. At that point, is Rust buying you anything for how much it's going to get in your way? I really don't see an advantage to Rust when operating at these kinds of low levels.

We have concrete data on this at this point, and it’s just not true that, even in these sorts of low level programs, everything ends up unsafe.

And beyond that, rust has many features that are useful separate from memory safety.

https://asahilinux.org/2022/11/tales-of-the-m1-gpu/ being just one example.

That said I have no opinion if they should write this driver in Rust or not, I simply do not know about the details. But in general, “it’s too low level and so tons of unsafe and so therefore Rust is useless” is at least arguable, if not just fully incorrect, as a general point.

Re: In OpenZFS and Btrfs, everyone was just guessing

#47

Post by calamari4065 »

mghardal wrote: post #38
onetimeuse92304 wrote: post #25

Earlier quoted context omitted.

The hardest bug I have ever worked with was an embedded device losing all data on the flash chip. But the issue was, this was happening even after we removed all instructions to delete data from the flash. The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness…

Which mcu are you using ? The possibility of noise becoming a flash command is practically 0. Also too much noise will more importantly corrupt you ram. Flash can be sensitive to noise and that can cause bit flips.

>The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died

That's one failure per 16,800 hours, or 700 days. That's pretty goddamn rare.

Re: In OpenZFS and Btrfs, everyone was just guessing

#48

Post by throw0101b »

solarkraft wrote: post #44
emaste wrote: post #35

Earlier quoted context omitted.

This is a longstanding tradition from FreeBSD. A list of our commit message trailers: https://docs.freebsd.org/en/articles/committers-guide/#_incl... . "Sponsored by:" search in FreeBSD commit messages: https://freshbsd.org/?q=%22Sponsored+by%3A%22

I like it! Seems like a great way to incentivise companies to contribute.

> I like it! Seems like a great way to incentivise companies to contribute.

It also helps with lawsuits, which was a major thing right at the beginning of history of BSD.

* https://en.wikipedia.org/wiki/UNIX_System_Laboratories,_Inc.....

Linux later had the lawsuit issue with SCO and IBM and tracking where certain things came from (which was not helped by the fact that Linus Torvalds refused to use source code tracking for the longest time (later taking up Bitkeeper, and later developing git)).

* https://en.wikipedia.org/wiki/SCO–Linux_disputes

Re: In OpenZFS and Btrfs, everyone was just guessing

#49

Post by Wowfunhappy »

OpenZFS is on an exceedingly short list of software in my life that I actually trust to do what it's supposed to. Where most software just up and falls over every so often, ZFS chugs along, day in and day out.

Then ZFS has one bug and everyone starts acting like the sky is falling. And sure, it was a bad bug, but you had to be pretty unlucky to trigger it, and it was present in ZFS for all of a month.

Meanwhile, people's Google Drives are apparently forgetting the last the eight months of changes.

OpenZFS is fine.

Re: In OpenZFS and Btrfs, everyone was just guessing

#50

Post by aidenn0 »

emaste wrote: post #36
dmpk2k wrote: post #16

While I am firmly in the ZFS camp, my feeling is that there has indeed been a gradual slide in the disciplined development of ZFS. After some initial bumps in brand new software, the Sun kernel group did a good job of avoiding corruption. Once Sun was absorbed by Oracle, OSS ZFS moved into illumos, who were overall quite good at doing the same (although they had less resources to play with). OpenZFS brought ZFS to th…

Note that the bug that is the topic of discussion here predates OpenZFS. Whether or not there has been a slide in disciplined development in OpenZFS, this bug does not support that assertion.

Since the topic is whether or not the current OZFS developers understand what is going on well enough to reliably fix the bug, I think it still applies.
Post reply on HN