Orphaning bcachefs-tools in Debian • Hacker News Board

Re: Orphaning bcachefs-tools in Debian

#101

Post by cesarb » Sat, Aug 31, 2024, 2:59 PM UTC

brnt wrote: post #96

koverstreet wrote: post #74

Earlier quoted context omitted.

This really isn't about bindgen. Any time a dependency is changed, the package version should be bumped and retested. This is a massive improvement in reproducibility, bisectability and our ability to QA. Before this model, bisecting breakage that was a result of a change in library 'a' breaking something in consumer 'b' was effectively impossible; now it is. What Debian and Fedora have been doing is a big step backw…

> What Debian and Fedora have been doing is a big step backwards, this whole 'unbundling of dependencies' (that were going to be statically linked anyways) needs to die. They, and most others, have been doing that since their respectively beginnings, and for good reasons: everybody expects distros to fix security issues, which is greatly aided by ensuring an-as-small-as-necesary dep tree and thus a single version of…

> They, and most others, have been doing that since their respectively beginnings

As I explained on another comment of mine (https://news.ycombinator.com/item?id=41409199), it's not since their beginning, but since a particular zlib security incident which showed them the risks of vendored libraries. Any discussion of vendoring policy which does not consider that incident is necessarily incomplete.

Re: Orphaning bcachefs-tools in Debian

#102

Post by BoingBoomTschak » Sat, Aug 31, 2024, 2:59 PM UTC

JonChesterfield wrote: post #3

This would be a good one for Rust enthusiasts to weigh in on. The issue raised is that some program written in rust insists on a very specific version of various dependencies. If other people change the metadata, it builds and seems to run ok with different versions. (Developer on reddit clarifies that it builds and does the wrong thing, and recommends dropping Debian as a solution). Linux likes to pack a finite set…

Gentoo has SLOTs and they work well. But for Go and Rust, I think they chose the vendoring path because it's just too damn Don Quixotesque to try to mirror their NPM tier package repositories.

Re: Orphaning bcachefs-tools in Debian

#103

Post by koverstreet » Sat, Aug 31, 2024, 3:01 PM UTC

smashed wrote: post #81

koverstreet wrote: post #43

I would rather something not be packaged at all than packaged badly; this whole experience reads like a lesson in what not to do. What the author left out in his blog post is that I specifically explained why this was going to be an issue and what was going to happen when he and I first talked about Debian packaging; and why for a package that needs to work correctly for the system to boot (fsck, mount helper, encryp…

> I would rather something not be packaged at all than packaged badly; this whole experience reads like a lesson in what not to do. You might consider adding a big warning on your official documentation about unsupported distribution packages. Add links to relevant issue tracker/bug reports or mailing list discussions saying until So and so issue are resolved, the official statement is that distribution package is un…

nah, I'm not taking the fight to Jörg Schilling levels :)

Re: Orphaning bcachefs-tools in Debian

#104

Post by ralferoo » Sat, Aug 31, 2024, 3:02 PM UTC

_ph_ wrote: post #95

ralferoo wrote: post #80

Earlier quoted context omitted.

It's very clear from that thread that he doesn't understand the purpose of the stable branch. It doesn't mean "stable" as in "the best possible experience", it means it as in "this code has been tested for a long period of time with no serious defects found" so that when the stable branch is promoted to release, everything has undergone a long testing period by a broad user base. If there is a defect found, the chang…

It seems to be a difficult situation: he has bug fixes against the version in the stable kernel for bugs which haven't been reported. I can see both perspectives: on stable you don't want to do development, but also you want all bugfixes you can get. I can also see the point of Linus, who wants just to add bug fixes and to minimize the risk of introducing new bugs. Considering that Kent himself warns against general…

I agree. If the author himself is telling people it's not ready for production, it doesn't really matter what bugs this code has, unless it affects other subsystems or is a dangerous regression from the previous stable release.

If the bug it's fixing was already in the current release branch, wasn't noticed before and has only shown up now late in the stable branch lifetime, then it definitely doesn't seem like something that needs an urgent fix.

Re: Orphaning bcachefs-tools in Debian

#105

Post by Twirrim » Sat, Aug 31, 2024, 3:03 PM UTC

yshui wrote: post #41

This is pretty dumb on Debian's part. First of all I don't understand why they insist crate dependencies must be pulled from their repository. They are just source code, not built binary. AFAIK there is no other distro that does this, what they do is that they would download crates from crates.io (`cargo vendor` is a command that does this automatically) and build against that. Arch does this, Gentoo does this, NixOS…

>AFAIK there is no other distro that does this, what they do is that they would download crates from crates.io (`cargo vendor` is a command that does this automatically) and build against that.

Note your examples are all bleeding edge / rolling distributions. Debian and the non-bleeding edge distributions go a different route and focus on reproduce-ability and security, among other things.

With the "get from crates.io" route, if someone compromises/hijacks a crate upstream, you're in trouble, immediately. By requiring vendoring of sources, that requires at least some level of manual actions by maintainers to get that compromised source in to the debian repositories to then be distributed out to the users. As you get in towards distributions like RHEL, they get even more cautious on this front.

Re: Orphaning bcachefs-tools in Debian

#106

Post by jonhohle » Sat, Aug 31, 2024, 3:04 PM UTC

yshui wrote: post #41

This is pretty dumb on Debian's part. First of all I don't understand why they insist crate dependencies must be pulled from their repository. They are just source code, not built binary. AFAIK there is no other distro that does this, what they do is that they would download crates from crates.io (`cargo vendor` is a command that does this automatically) and build against that. Arch does this, Gentoo does this, NixOS…

I’d consider the issue to be the opposite. Why does every programming language now have a package manager and all of the infrastructure around package management rather than rely on the OS package manager? As a user I have to deal with apt, ports, pkg, opkg, ipkg, yum, flatpak, snap, docker, cpan, ctan, gems, pip, go modules, cargo, npm, swift packages, etc., etc., which all have different opinions of how and where to package files.

On packaged operating systems (Debian, FreeBSD) - you have the system’s package manager to deal with (apt, pkg respectively). I can have an offline snapshot of _all_ packages that can be mirrors from one place.

IMHO, every programming language having its own package system is the weird thing.

Re: Orphaning bcachefs-tools in Debian

#107

Post by brnt » Sat, Aug 31, 2024, 3:06 PM UTC

cesarb wrote: post #101

brnt wrote: post #96

Earlier quoted context omitted.

> What Debian and Fedora have been doing is a big step backwards, this whole 'unbundling of dependencies' (that were going to be statically linked anyways) needs to die. They, and most others, have been doing that since their respectively beginnings, and for good reasons: everybody expects distros to fix security issues, which is greatly aided by ensuring an-as-small-as-necesary dep tree and thus a single version of…

> They, and most others, have been doing that since their respectively beginnings As I explained on another comment of mine ( https://news.ycombinator.com/item?id=41409199 ), it's not since their beginning, but since a particular zlib security incident which showed them the risks of vendored libraries. Any discussion of vendoring policy which does not consider that incident is necessarily incomplete.

How long ago was that? And did they quantify the problem?

Re: Orphaning bcachefs-tools in Debian

#108

Post by aseipp » Sat, Aug 31, 2024, 3:08 PM UTC

rlpb wrote: post #32

cogman10 wrote: post #18

Earlier quoted context omitted.

It should be noted that, particularly for rust, it's a silly policy as rust does not support dynamic linking of rust libraries. All rust applications are statically linked. So trying to force all projects using foo onto the same version of foo is just a huge headache with no real benefit to anything.

> So trying to force all projects using foo onto the same version of foo is just a huge headache with no real benefit to anything. It creates a security nightmare. Distributions expect to be able to security patch without waiting for laggard upstreams to bump their pinned dependency versions. If the ecosystem insists that only upstream-sanctioned versions are acceptable, then that runs contrary to this expectation, b…

Not all distributions suffer from this, because they have the tooling to patch all crate versions given a name. Compile times are actually the worst part but can be mitigated. But these distros also fundamentally accept the language's compilation model in question, so they have to develop solutions if they want to supply software to their users rather than make up excuses. Most distros don't like vendoring, for reasons. But many code maintainers do like it, for reasons. So it is what it is.

There are also more tools than ever to do things like correlate versions of OSS packages with reported vulnerabilities e.g. Google's osv.dev. 95% or more of Rust vendoring isn't actually "vendoring" in the sense you have to figure anything out. The dependencies are literally written in the lock file with the version and that can be traced back to a Git hash automatically in almost all cases. Some crates that bind C libraries might require this work, where you have to look at the code to see "What version of the code did the `cp -R` into here?" but they are practically limited in number, and detectable. You aren't doing any detective work for 98% of this stuff. It can be scaled pretty well with a few (good) heuristics.

Rust even has the advantage that large, vast amounts of code is actually statically analyzable under known sets of configurations/features. In the future I suspect a tool like `govulncheck` for Rust will be developed, making accurate vuln tracing even easier.

While I'm admittedly biased because I work on a distro where this problem isn't as directly relevant, my answer is that in 2024 if you don't have the tooling to do this stuff, it's absolutely on you. Whether as a policy decision or a technical limitation, whatever.

Re: Orphaning bcachefs-tools in Debian

#109

Post by htpart » Sat, Aug 31, 2024, 3:09 PM UTC

peppermint_gum wrote: post #30

>not even considering some hostile emails that I recently received from the upstream developer or his public rants on lkml and reddit It feels like whenever the author of bcachefs comes up, it's always because of some drama. Just the other day he clashed with Linus Torvalds: https://lore.kernel.org/lkml/CAHk-=wj1Oo9-g-yuwWuHQZU8v=VAsB... My reading is that he's very passionate, so he wants to "move fast and break thi…

That's a normal LKML conversation. Nowhere do I see actual bugs pointed out, in fact the only testimony by Carl said that bcachefs has been quite stable so far.

This is just about following procedure. There are people who follow procedure and introduce many bugs, and people who don't but write perfect software.

The bcachefs author cautiously marks bcachefs as unstable, which is normal for file systems. The only issue here is that the patch touched other areas, but in the kernel development model Linus is free not to pull it.

Re: Orphaning bcachefs-tools in Debian

#110

Post by ploxiln » Sat, Aug 31, 2024, 3:10 PM UTC

foobarqux wrote: post #82

ploxiln wrote: post #70

Earlier quoted context omitted.

Shouldn't the filesystem utilities be base-system packages, though? Like e2fsprogs, xfsprogs ... I guess rust just isn't appropriate for base-system packages, because the dependencies move too fast, and so rust components need to depend on different specific versions of all dependencies, and are "horrifically out-of-date" in just a few months ... and rust developers wouldn't stoop so low as to recognize that e.g. bin…

I'm not that familiar with bcachefs-tools and was speaking more generally but I thought bcachefs-tools wasn't critical for normal booting (On Ubuntu 24.04 nothing seems to depend on that package and it seems optional. The same for btrfs-tools). Some other comments suggested you need it to mount a degraded file system but it seems to me that is a case where either 1. your system can't boot and you need external recove…

IIRC the brokenness was simply due to a bug in an upstream release of bindgen, which "was quickly fixed upstream, but as far as I know it's still broken in Debian" (which must have some reason of not updating bindgen again ...) so it wasn't some subtle incompatibility or modification to bcachefs-tools IMHO, it was just bindgen.