Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bcachefs Goes to "Externally Maintained"

lwn.net

Post reply on HN
Print view View original

101–110 of 400 posts

Re: Bcachefs Goes to "Externally Maintained"

#101

Post by koverstreet »

tarruda wrote: post #46

Since the existing bcachefs driver will not be removed, and the problem is the bcachefs developer not following the rules, I wonder if someone else could take on the role of pulling bcachefs changes into the mainline, while also following the merge window rules.

No, the problem wasn't following the rules.

The patch that kicked off the current conflict was the 'journal_rewind' patch; we recently (6.15) had the worst bug in the entire history upstream - it was taking out entire subvolumes.

The third report got me a metadata dump with everything I needed to debug the issue, thank god, and now we have a great deal of hardening to ensure a bug like this can never happen again. Subsequently, I wrote new repair code, which fully restored the filesystem of the 3rd user hit by the bug (first two had backups).

Linus then flipped out because it was listed as a 'feature' in the pull request; it was only listed that way to make sure that users would know about it if they were affected by the original bug and needed it. Failure to maintain your data is always a bug for a filesystem, and repair code is a bugfix.

In the private maintainer thread, and even in public, things went completely off the rails, with Linus and Ted basically asserting that they knew better than I do which bcachefs patches are regression risks (seriously), and a page and a half rant from Linus on how he doesn't trust my judgement, and a whole lot more.

There have been many repeated arguments like this over bugfixes.

The thing is, since then I started perusing pull requests from other subsystems, and it looks like I've actually been more conservative with what I consider a critical bugfix (and send outside the merge window) than other subsystems. The _only_ thing that's been out of the ordinary with bcachefs has been the volume of bugfixes - but that's exactly what you'd expect to see from a new filesystem that's stabilizing rapidly and closing out user bug reports - high volume of pure bugfixing is exactly what you want to see.

So given that, I don't think having a go-between would solve anything.

Re: Bcachefs Goes to "Externally Maintained"

#102

Post by koverstreet »

ajross wrote: post #95
nh2 wrote: post #90

Earlier quoted context omitted.

How can I, with dm/lvm: * For some detected corruption, be told directly which files are affected? * Get filesystem level snapshots that are guaranteed to be consistent in the way ZFS and CephFS snapshots guarantee?

On urging from tptacek I'll take that seriously and not as flame: 1. This is misunderstanding how device corruption works. It's not and can't ever be limited to "files". (Among other things: you can lose whole trees if a directory gets clobbered, you'd never even be able to enumerate the "corrupted files" at all!). All you know (all you can know) is that you got a success and that means the relevant data and metadata…

1 - you absolutely can and should walk reverse mappings in the filesystem so that from a corrupt block you can tell the user which file was corrupted.

In the future bcachefs will be rolling out auxiliary dirent indices for a variety of purposes, and one of those will be to give you a list of files that have had errors detected by e.g. scrub (we already generally tell you the affected filename in error messages)

2 - No, metadata robustness absolutely varies across filesystems.

From what I've seen, ext4 and bcachefs are the gold standard here; both can recover from basically arbitrary corruption and have no single points of failure.

Other filesystems do have single points of failure (notably btree roots), and btrfs and I believe ZFS are painfully vulnerable to devices with broken flush handling. You can blame (and should) blame the device and the shitty manufacturers, but from the perspective of a filesystem developer, we should be able to cope with that without losing the entire filesystem.

XFS is quite a bit better than btrfs, and I believe ZFS, because they have a ton of ways to reconstruct from redundant metadata if they lose a btree root, but it's still possible to lose the entire filesystem if you're very, very unlucky.

On a modern filesystem that uses b-trees, you really need a way of repairing from lost b-tree roots if you want your filesystem to be bulletproof. btrfs has 'dup' mode, but that doesn't mean much on SSDs given that you have no control over whether your replicas get written to the same erase unit.

Reiserfs actually had the right idea - btree node scan, and reconstruct your interior nodes if necessary. But they gave that approach a bad name; for a long time it was a crutch for a buggy b-tree implementation, and they didn't seed a filesystem specific UUID into the btree node magic number like bcachefs does, so it could famously merge a filesystem from a disk image with the host filesystem.

bcachefs got that part right, and also has per-device bitmaps in the superblock for 'this range of the device has btree nodes' so it's actually practical even if you've got a massive filesystem on spinning rust - and it was introduced long after the b-tree implementation was widely deployed and bulletproof.

Re: Bcachefs Goes to "Externally Maintained"

#103

Post by koverstreet »

tux3 wrote: post #60
immibis wrote: post #55

Earlier quoted context omitted.

That person would be accountable to Linus, but not to Kent.

Unfortunately, there's also nothing they can do if Kent says no. Say there's a disagreement on a patch that touches something outside fs/bcachefs, that person can't exactly write their own patches incorporating the feedback. They're not going to fork and maintain their own patches. They'd be stuck between a rock and a hard place, and that gets us back to a deadlock. The issue is that I have never seen Kent back down…

The key thing here is I've never challenged Linus's authority on patches outside fs/bcachefs/; I've quietly respun pull requests for that, on more than one occasion.

The point of contention here was a patch within fs/bcachefs/, which was repair code to make sure users didn't lose data.

If we can't have clear boundaries and delineations of responsibility, there really is no future for bcachefs in the kernel; my core mission is a rock solid commitment to reliability and robustness, including being responsive to issues users hit, and we've seen repeatedly that the kernel process does not share those priorities.

Re: Bcachefs Goes to "Externally Maintained"

#104

Post by nirava »

koverstreet wrote: post #101
tarruda wrote: post #46

Since the existing bcachefs driver will not be removed, and the problem is the bcachefs developer not following the rules, I wonder if someone else could take on the role of pulling bcachefs changes into the mainline, while also following the merge window rules.

No, the problem wasn't following the rules. The patch that kicked off the current conflict was the 'journal_rewind' patch; we recently (6.15) had the worst bug in the entire history upstream - it was taking out entire subvolumes. The third report got me a metadata dump with everything I needed to debug the issue, thank god, and now we have a great deal of hardening to ensure a bug like this can never happen again. Su…

To list down the current state of things:

1. Regardless of whether correct or not, it's Linus that decides what's a feature and what's not in Linux. Like he has for the last however many decades. Repair code is a feature if Linus says it is a feature.

2. Being correct comes second to being agreeable in human-human interactions. For example, dunking on x file system does not work as a defense when the person opposite you is a x file system maintainer.

3. rules are rules, and generally don't have to be "correct" to be enforced in an organization

I think your perceived "unfairness" might make sense if you just thought of these things as un-workaroundable constraints, Just like the fact that SSDs wear out over time.

Re: Bcachefs Goes to "Externally Maintained"

#105

Post by koverstreet »

nirava wrote: post #104
koverstreet wrote: post #101

Earlier quoted context omitted.

No, the problem wasn't following the rules. The patch that kicked off the current conflict was the 'journal_rewind' patch; we recently (6.15) had the worst bug in the entire history upstream - it was taking out entire subvolumes. The third report got me a metadata dump with everything I needed to debug the issue, thank god, and now we have a great deal of hardening to ensure a bug like this can never happen again. Su…

To list down the current state of things: 1. Regardless of whether correct or not, it's Linus that decides what's a feature and what's not in Linux. Like he has for the last however many decades. Repair code is a feature if Linus says it is a feature. 2. Being correct comes second to being agreeable in human-human interactions. For example, dunking on x file system does not work as a defense when the person opposite…

When rules and authority start to take precedence over making sure things work, things have gone off the rails and we're not doing engineering anymore.

Re: Bcachefs Goes to "Externally Maintained"

#106

Post by koverstreet »

arp242 wrote: post #84
bornfreddy wrote: post #65

Earlier quoted context omitted.

Being an outsider to this whole scene, the whole thread reads very differently to me. Kent seems very patient in explaining his position (and frustrations arising from other people introducing bugs to his code) and the kernel & debian folks are performing a smearing campaign instead of replying to what I see are genuine problems in the process. As an example, the quotes that are referenced by user paravoid are, imho,…

Kent brings up Debian himself, unprompted. This is one of the problems: Kent is frequently unable to accept that things don't go his way. He will keep bringing it up again and again and he just grinds people down with it. If you see just one bit of it then it may seem somewhat reasonable, but it's really not because this is the umpteenth time this exact discussion is happening and it's groundhog day once again. This…

You do realize that data integrity issues are not "live and let live" type things, right?

And there's a real connection to the issue that sparked all this drama in the kernel and the Debian drama: critical system components (the kernel, the filesystem, and others) absolutely need to be able to get bugfixes in a timely manner. That's not optional.

With Debian, we had a package maintainer who decided that unbundling Rust dependencies was more important than getting out updates, and then we couldn't get a bugfix out for mount option handling. This was a non-issue for every other distro with working processes because the bug was fixed in a few days, but a lot of Debian users weren't able to mount in degraded mode and lost access to their filesystems.

In the kernel drama, Linus threw a fit over a repair code to recover from a serious bug and make sure users didn't lose data, and he's repeatedly picked fights over bugfixes (and even called pushing for getting bugfixes out "whining" in the past).

There are a lot of issues that there can be give and take on, but getting fixes out in a timely manner is just part of the baseline set of expectations for any serious project.

Re: Bcachefs Goes to "Externally Maintained"

#107

Post by tux3 »

koverstreet wrote: post #103
tux3 wrote: post #60

Earlier quoted context omitted.

Unfortunately, there's also nothing they can do if Kent says no. Say there's a disagreement on a patch that touches something outside fs/bcachefs, that person can't exactly write their own patches incorporating the feedback. They're not going to fork and maintain their own patches. They'd be stuck between a rock and a hard place, and that gets us back to a deadlock. The issue is that I have never seen Kent back down…

The key thing here is I've never challenged Linus's authority on patches outside fs/bcachefs/; I've quietly respun pull requests for that, on more than one occasion. The point of contention here was a patch within fs/bcachefs/, which was repair code to make sure users didn't lose data. If we can't have clear boundaries and delineations of responsibility, there really is no future for bcachefs in the kernel; my core m…

You may be right, but I think looking at it from a lens of who has authority and can impose their decision is still illustrating the point I'm trying to make.

To some extent drawing clear boundaries is good as a last resort when people cannot agree, but it can't be the main way to resolve disagreements. Thinking in terms of who owns what and has the final say is not the same as trying to understand the requirements from the other side to find a solution that works for everyone.

I don't think the right answer is to blindly follow whatever Linus or other people say. I don't mean you should automatically back down without technical reasons, because authority says so. But I notice I can't remember an email where concessions where made, or attemps to find a middle grounds by understanding the other side. Maybe someone can find counterexamples.

But this idea of using ownership to decide who has more authority and can impose their vision, that can't be the only way to collaborate. It really is uncompromising.

Re: Bcachefs Goes to "Externally Maintained"

#108

Post by yencabulator »

ffsm8 wrote: post #100
BoredPositron wrote: post #96

Earlier quoted context omitted.

Grounded? Not offensive? https://lore.kernel.org/lkml/CAHk-=wiLE9BkSiq8F-mFW5NOtPzYrt... https://lore.kernel.org/all/citv2v6f33hoidq75xd2spaqxf7nl5wb...

The first one is by Linus? And his replies (at least the ones I read) are - to me- less aggressive then the rest of the mails in that chain The second has one offensive remark: > Get your head examined. And get the fuck out of here with this shit. which I thought he admitted was out of line and - said sorry for. Or do I misremember? I admit once again, I'm still completely uninvolved and merely saw it play out on the…

If you read his replies downthread from that, Kent seems to be going through a lot of effort to not apologize, in any form, and prefers talking about how other people were mean to him.

I had high hopes for bcachefs. sigh

Re: Bcachefs Goes to "Externally Maintained"

#109

Post by koverstreet »

tux3 wrote: post #107
koverstreet wrote: post #103

Earlier quoted context omitted.

The key thing here is I've never challenged Linus's authority on patches outside fs/bcachefs/; I've quietly respun pull requests for that, on more than one occasion. The point of contention here was a patch within fs/bcachefs/, which was repair code to make sure users didn't lose data. If we can't have clear boundaries and delineations of responsibility, there really is no future for bcachefs in the kernel; my core m…

You may be right, but I think looking at it from a lens of who has authority and can impose their decision is still illustrating the point I'm trying to make. To some extent drawing clear boundaries is good as a last resort when people cannot agree, but it can't be the main way to resolve disagreements. Thinking in terms of who owns what and has the final say is not the same as trying to understand the requirements f…

> To some extent drawing clear boundaries is good as a last resort when people cannot agree, but it can't be the main way to resolve disagreements. Thinking in terms of who owns what and has the final say is not the same as trying to understand the requirements from the other side to find a solution that works for everyone.

Agreed 100%. In an ideal world, we'd be sitting down together, figuring out what our shared priorities are, and working from there.

Unfortunately, that hasn't been possible, and I have no idea what Linus's priorities except that they definitely aren't a bulletproof filesystem and safeguarding user data; his response to journal_rewind demonstrated that quite definitively.

So that's where we're at, and given the history with other local filesystems I think I have good reason not to concede. I don't want to see bcachefs run off the rails, but given all the times I've talked about process and the way I'm doing things I think that's exactly what would happen if I started conceding on these points. It's my life's work, after all.

You'd think bcachefs's track record (e.g. bug tracker, syzbot) and the response it gets from users would be enough, but apparently not, sadly. But given the way the kernel burns people out and outright ejects them, not too surprising.

Re: Bcachefs Goes to "Externally Maintained"

#110

Post by khimaros »

ajross wrote: post #63
anon-3988 wrote: post #36

Earlier quoted context omitted.

> Filesystems are a solved problem. If ZFS disappeared from the world today... really who would even care? Only those of us still around trying to shout on the internet. Yeah nah, have you tried processing terabytes of data every day and storing them? It gets better now with DDR5 but bit flips do actually happen.

And once more, you're positing the lack of a feature that is available and very robust (c.f. "yell on the internet" vs. "discuss solutions to a problem"). You don't need your filesystem to integrate checksumming when dm/lvm already do it for you.

i'm not one for internet arguments and really just want solutions. maybe you could point me at the details for a setup that worked for you?

based on my own testing, dm has a lot of footguns and, with some kernels, as little as 100 bytes of corruption to the underlying disk could render a dm-integrity volume completely unusable (requiring a full rebuild) https://github.com/khimaros/raid-explorations

Post reply on HN