Bcachefs Goes to "Externally Maintained" • Hacker News Board

Re: Bcachefs Goes to "Externally Maintained"

#111

Post by magicalhippo » Sun, Aug 31, 2025, 12:32 AM UTC

koverstreet wrote: post #102

ajross wrote: post #95

Earlier quoted context omitted.

On urging from tptacek I'll take that seriously and not as flame: 1. This is misunderstanding how device corruption works. It's not and can't ever be limited to "files". (Among other things: you can lose whole trees if a directory gets clobbered, you'd never even be able to enumerate the "corrupted files" at all!). All you know (all you can know) is that you got a success and that means the relevant data and metadata…

1 - you absolutely can and should walk reverse mappings in the filesystem so that from a corrupt block you can tell the user which file was corrupted. In the future bcachefs will be rolling out auxiliary dirent indices for a variety of purposes, and one of those will be to give you a list of files that have had errors detected by e.g. scrub (we already generally tell you the affected filename in error messages) 2 - N…

> XFS is quite a bit better than btrfs, and I believe ZFS, because they have a ton of ways to reconstruct from redundant metadata if they lose a btree root

As I understand it ZFS also has a lot of redundant metatdata (copies=3 on anything important), and also previous uberblocks[1].

In what way is XFS better? Genuine question, not really familiar with XFS.

[1]: https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSMetadata...

Re: Bcachefs Goes to "Externally Maintained"

#112

Post by yencabulator » Sun, Aug 31, 2025, 12:35 AM UTC

yjftsjthsd-h wrote: post #97

williamstein wrote: post #43

Earlier quoted context omitted.

So do I and BTRFS is extremely good these days. It's also much faster than ZFS at mounting a disk with a large number of filesystems (=subvolumes), which is critical for building certain types of fileservers at scale. In contrast, ZFS scales horribly as the number of filesystems increases, where btrfs seems to be O(1). btrfs's quota functionality is also much better than it used to be (and very flexible), after all t…

> It's also much faster than ZFS at mounting a disk with a large number of filesystems (=subvolumes), which is critical for building certain types of fileservers at scale. Now you've piqued my curiosity; what uses that many filesystems/subvolumes? (Not an attack; I believe you, I'm just trying to figure out where it comes up)

As far as I understand, a core use case at Meta was build system workers starting with prepopulated state and being able to quickly discard the working tree at the end of the build. CoW is pretty sweet for that.

Re: Bcachefs Goes to "Externally Maintained"

#113

Post by tux3 » Sun, Aug 31, 2025, 12:37 AM UTC

koverstreet wrote: post #109

tux3 wrote: post #107

Earlier quoted context omitted.

You may be right, but I think looking at it from a lens of who has authority and can impose their decision is still illustrating the point I'm trying to make. To some extent drawing clear boundaries is good as a last resort when people cannot agree, but it can't be the main way to resolve disagreements. Thinking in terms of who owns what and has the final say is not the same as trying to understand the requirements f…

> To some extent drawing clear boundaries is good as a last resort when people cannot agree, but it can't be the main way to resolve disagreements. Thinking in terms of who owns what and has the final say is not the same as trying to understand the requirements from the other side to find a solution that works for everyone. Agreed 100%. In an ideal world, we'd be sitting down together, figuring out what our shared pr…

Fair enough. As someone who has lost filesystems to bugs and files to corrupted blocks, I definitely appreciate the work you've done on repair and reliability.

I think there's room to have your cake and eat it too, but I certainly can't blame you for caring about quality, that much is sure.

Re: Bcachefs Goes to "Externally Maintained"

#114

Post by AaronFriel » Sun, Aug 31, 2025, 12:39 AM UTC

wtallis wrote: post #62

AaronFriel wrote: post #17

Earlier quoted context omitted.

Respectfully to the maintainers: How can this be a stable filesystem if parity is unstable and risks data loss? How has this been allowed to happen? It just seems so profoundly unserious to me.

Does the whole filesystem need to be marked as unstable if it has a single experimental feature? Is any other filesystem held to that standard?

Parity support in multi-disk arrays is older than I am, it's a fairly standard feature. btrfs doesn't support this without data loss risks after 17 years of development.

Re: Bcachefs Goes to "Externally Maintained"

#115

Post by williamstein » Sun, Aug 31, 2025, 12:41 AM UTC

yjftsjthsd-h wrote: post #97

williamstein wrote: post #43

Earlier quoted context omitted.

So do I and BTRFS is extremely good these days. It's also much faster than ZFS at mounting a disk with a large number of filesystems (=subvolumes), which is critical for building certain types of fileservers at scale. In contrast, ZFS scales horribly as the number of filesystems increases, where btrfs seems to be O(1). btrfs's quota functionality is also much better than it used to be (and very flexible), after all t…

> It's also much faster than ZFS at mounting a disk with a large number of filesystems (=subvolumes), which is critical for building certain types of fileservers at scale. Now you've piqued my curiosity; what uses that many filesystems/subvolumes? (Not an attack; I believe you, I'm just trying to figure out where it comes up)

It can be useful to create a file server with one filesystem/subvolume per user, because each user has their own isolated snapshots, backups via send/recv are user-specific, quotas are easier, etc. If you only have a few hundred users, ZFS is fine. But what if you have 100,000 users? Then just doing "zpool import" would take hours, whereas mounting a btrfs filesystem with 100,000 subvolumes takes a seconds. This complexity difference was a show stopper for me to architect a certain solution on top of ZFS, despite me personally loving ZFS and having used it for a long time. The btrfs commands and UX are really awkward (for me) compared to ZFS, but btrfs is extremely efficient at some things where ZFS just falls down.

The main criticism in this thread about btrfs involves multidisk setups, which aren't relevant for me, since I'm working on cloud systems and disk storage is abstracted away as a single block device.

Re: Bcachefs Goes to "Externally Maintained"

#116

Post by williamstein » Sun, Aug 31, 2025, 12:42 AM UTC

williamstein wrote: post #115

yjftsjthsd-h wrote: post #97

Earlier quoted context omitted.

> It's also much faster than ZFS at mounting a disk with a large number of filesystems (=subvolumes), which is critical for building certain types of fileservers at scale. Now you've piqued my curiosity; what uses that many filesystems/subvolumes? (Not an attack; I believe you, I'm just trying to figure out where it comes up)

It can be useful to create a file server with one filesystem/subvolume per user, because each user has their own isolated snapshots, backups via send/recv are user-specific, quotas are easier, etc. If you only have a few hundred users, ZFS is fine. But what if you have 100,000 users? Then just doing "zpool import" would take hours, whereas mounting a btrfs filesystem with 100,000 subvolumes takes a seconds. This comp…

Incidentally, the application I'm reworking to use btrfs is cocalc.com. One of our main use cases is distributed assignments to students in classes, as part of the course management functionality. Imagine a class with 1500 students all getting an exact copy of a 50 MB folder, which they'll edit a little bit, and then it will be collected. The copy-on-write functionality of btrfs is fantastic for this use case (both in speed and disk usage).

Also, the out-of-band deduplication for btrfs using https://github.com/Zygo/bees is very impressive and flexible, in a way that ZFS just doesn't match.

Re: Bcachefs Goes to "Externally Maintained"

#117

Post by koverstreet » Sun, Aug 31, 2025, 12:47 AM UTC

magicalhippo wrote: post #111

koverstreet wrote: post #102

Earlier quoted context omitted.

1 - you absolutely can and should walk reverse mappings in the filesystem so that from a corrupt block you can tell the user which file was corrupted. In the future bcachefs will be rolling out auxiliary dirent indices for a variety of purposes, and one of those will be to give you a list of files that have had errors detected by e.g. scrub (we already generally tell you the affected filename in error messages) 2 - N…

> XFS is quite a bit better than btrfs, and I believe ZFS, because they have a ton of ways to reconstruct from redundant metadata if they lose a btree root As I understand it ZFS also has a lot of redundant metatdata (copies=3 on anything important), and also previous uberblocks[1]. In what way is XFS better? Genuine question, not really familiar with XFS. [1]: https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSMeta…

I can't speak with any authority on ZFS, I know its structure the least out of all the major filesystems.

I do a ton of reading through forums gathering user input, and lots of people chime in with stories of lost filesystems. I've seen reports of lost filesystems with ZFS and I want to say I've seen them at around the same frequency of XFS; both are very rare.

My concern with ZFS is that they seem to have taken the same "no traditional fsck" approach as btrfs, favoring entirely online repair. That's obviously where we all want to be, but that's very hard to get right, and it's been my experience that if you prioritize that too much you miss the "disaster recovery" scenarios, and that seems to be what's happened with ZFS; I've read that if your ZFS filesystem is toast you need to send it to a data recovery service.

That's not something I would consider acceptable, fsck ought to be able to do anything a data recovery service would do, and for bcachefs it does.

I know the XFS folks have put a ton of outright paranoia into repair, including full on disaster recovery scenarios. It can't repair in scenarios where bcachefs can - but on the other hand, XFS has tricks that bcachefs doesn't, so I can't call bcachefs unequivocally better; we'd need to wait for more widespread usage and a lot more data.

Re: Bcachefs Goes to "Externally Maintained"

#118

Post by magicalhippo » Sun, Aug 31, 2025, 1:03 AM UTC

koverstreet wrote: post #109

tux3 wrote: post #107

Earlier quoted context omitted.

You may be right, but I think looking at it from a lens of who has authority and can impose their decision is still illustrating the point I'm trying to make. To some extent drawing clear boundaries is good as a last resort when people cannot agree, but it can't be the main way to resolve disagreements. Thinking in terms of who owns what and has the final say is not the same as trying to understand the requirements f…

> To some extent drawing clear boundaries is good as a last resort when people cannot agree, but it can't be the main way to resolve disagreements. Thinking in terms of who owns what and has the final say is not the same as trying to understand the requirements from the other side to find a solution that works for everyone. Agreed 100%. In an ideal world, we'd be sitting down together, figuring out what our shared pr…

> Unfortunately, that hasn't been possible, and I have no idea what Linus's priorities except that they definitely aren't a bulletproof filesystem and safeguarding user data

Remarks like this come across as extremely patronizing, as you completely ignore what the other party says and instead project your own conclusions about the other persons motives and beliefs.

> his response to journal_rewind demonstrated that quite definitively

No, no it did not in any shape way or form do that. You had multiple other perfectly valid options to help the affected users besides getting that code shipped in the kernel there and then. Getting it shipped in the kernel was merely a convenience.

If bcachefs was established and stable it would be a different matter. But it's an experimental file system. Per definition data loss is to be expected, even if recovery is preferable.

Re: Bcachefs Goes to "Externally Maintained"

#119

Post by koverstreet » Sun, Aug 31, 2025, 1:13 AM UTC

magicalhippo wrote: post #118

koverstreet wrote: post #109

Earlier quoted context omitted.

> To some extent drawing clear boundaries is good as a last resort when people cannot agree, but it can't be the main way to resolve disagreements. Thinking in terms of who owns what and has the final say is not the same as trying to understand the requirements from the other side to find a solution that works for everyone. Agreed 100%. In an ideal world, we'd be sitting down together, figuring out what our shared pr…

> Unfortunately, that hasn't been possible, and I have no idea what Linus's priorities except that they definitely aren't a bulletproof filesystem and safeguarding user data Remarks like this come across as extremely patronizing, as you completely ignore what the other party says and instead project your own conclusions about the other persons motives and beliefs. > his response to journal_rewind demonstrated that qu…

No, bcachefs-tools wasn't an option because the right way to do this kind of repair is to first do a dry run test repair and mount, so you can verify with your eyes that everything is back as it should be.

If we had the fuse driver done that would have worked, though. Still not completely ideal because we're at the mercy of distros to make sure they're getting -tools updates out in a timely manner, they're not always as consistent with that as the kernel. Most are good, though).

Just making it available in a git repo was not an option because lots of bcachefs users are getting it from their distro kernel and have never built a kernel before (yes, I've had to help users with building kernels for the first time; it's slow and we always look for other options), and even if you know how, if your primary machine is offline the last thing you want to have to do is build a custom rescue image with a custom kernel.

And there was really nothing special about this than any other bugfix, besides needing to use a new option (which is also something that occasionally happens with hotfixes).

Bugs are just a fact of life, every filesystem has bugs and occasionally has to get hotfixes out quickly. It's just not remotely feasible or sane to be coming up with our own parallel release process for hotfixes.

Re: Bcachefs Goes to "Externally Maintained"

#120

Post by arp242 » Sun, Aug 31, 2025, 1:22 AM UTC

koverstreet wrote: post #106

arp242 wrote: post #84

Earlier quoted context omitted.

Kent brings up Debian himself, unprompted. This is one of the problems: Kent is frequently unable to accept that things don't go his way. He will keep bringing it up again and again and he just grinds people down with it. If you see just one bit of it then it may seem somewhat reasonable, but it's really not because this is the umpteenth time this exact discussion is happening and it's groundhog day once again. This…

You do realize that data integrity issues are not "live and let live" type things, right? And there's a real connection to the issue that sparked all this drama in the kernel and the Debian drama: critical system components (the kernel, the filesystem, and others) absolutely need to be able to get bugfixes in a timely manner. That's not optional. With Debian, we had a package maintainer who decided that unbundling Ru…

Look, I get where you're coming from. It's not unreasonable. I've said this before.

But there are also reasons why things are the way they are, and that is also not unreasonable. And at the end of the day: Linus is the boss. It really does come down to that. He has dozens of other subsystem maintainers to deal with and this is the process that works for him.

Similar stuff applies to Debian. Personally, I deeply dislike Debian's inflexible and outmoded policy and lack of pragmatism. But you know, the policy is the policy, and at some point you just need to accept that and work with it the best you can.

It's okay to make all the arguments you've made. It's okay to make them forcefully (within some limits of reason). It's not okay to keep repeating them again and again until everyone gets tired of it and seemingly just completely fail to listen to what people are sating. This is where you are being unreasonable.

I mean, you *can* do that, I guess, but look at where things are now. No one is happy with this – certainly not you. And it's really not a surprise, I already said this in November last year: "I wouldn't be surprised to see bcachefs removed from the kernel at some point".[1] To be clear: I didn't want that to happen – I think you've done great work with bcachefs and I really want it to succeed every which way. But everyone could see this coming from miles.

[1]: https://news.ycombinator.com/item?id=42225345