Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Orphaning bcachefs-tools in Debian

jonathancarter.org

Post reply on HN
Print view View original

71–80 of 224 posts

Re: Orphaning bcachefs-tools in Debian

#71

Post by hamandcheese »

rlpb wrote: post #32
cogman10 wrote: post #18

Earlier quoted context omitted.

It should be noted that, particularly for rust, it's a silly policy as rust does not support dynamic linking of rust libraries. All rust applications are statically linked. So trying to force all projects using foo onto the same version of foo is just a huge headache with no real benefit to anything.

> So trying to force all projects using foo onto the same version of foo is just a huge headache with no real benefit to anything. It creates a security nightmare. Distributions expect to be able to security patch without waiting for laggard upstreams to bump their pinned dependency versions. If the ecosystem insists that only upstream-sanctioned versions are acceptable, then that runs contrary to this expectation, b…

If the only way a security team can cope is with a single version policy, that sounds to me like a tooling problem.

Take Nixpkgs, for example. They have this concept called "default crate overrides" which their rust builder uses, which offers the capability to patch all versions of a crate. That's not to say that Nixpkgs does rust perfectly, but just an example of the sort of tooling a security team should have at their disposal.

Re: Orphaning bcachefs-tools in Debian

#72

Post by rfoo »

rlpb wrote: post #14
JonChesterfield wrote: post #3

This would be a good one for Rust enthusiasts to weigh in on. The issue raised is that some program written in rust insists on a very specific version of various dependencies. If other people change the metadata, it builds and seems to run ok with different versions. (Developer on reddit clarifies that it builds and does the wrong thing, and recommends dropping Debian as a solution). Linux likes to pack a finite set…

Maintenance is much more practical if everything in your dependency tree uses the same version of every dependency. Achieving this in practice is much of the work distribution maintainers have to do. Doing this has a couple of key advantages: 1. when a security vulnerability needs patching in a stable distribution release, the security team only have to patch one version instead of a dozen or more different versions…

No.

1. Rust is as hostile to dynamic libraries as glibc to static.

2. With everything static you have to rebuild every dependent on any security patch anyways. If you meant with multiple versions maintainers have to backport patches to multiple versions. Maybe don't backport at all? Some people appreciate having backports. Users of software written in Rust does NOT. So why bother doing backports for them?

3.

> a library that depends on dependencies A and B, both of which depend on X, can actually work properly

This will be detected during compile (since it's all static! sorry dynamic linking fans) and dealt with.

Re: Orphaning bcachefs-tools in Debian

#73

Post by sp1rit »

peppermint_gum wrote: post #30

>not even considering some hostile emails that I recently received from the upstream developer or his public rants on lkml and reddit It feels like whenever the author of bcachefs comes up, it's always because of some drama. Just the other day he clashed with Linus Torvalds: https://lore.kernel.org/lkml/CAHk-=wj1Oo9-g-yuwWuHQZU8v=VAsB... My reading is that he's very passionate, so he wants to "move fast and break thi…

[flagged]

Re: Orphaning bcachefs-tools in Debian

#74

Post by koverstreet »

haileys wrote: post #62
koverstreet wrote: post #53

Earlier quoted context omitted.

I already provide tarballs with vendored dependencies - I did this for Fedora, so that package builds wouldn't have to download anything. I'm not going to special case the bindgen bindings because this really isn't a bindgen specific issue - this could just as easily been any other dependency.

Speaking from practical experience, bindgen is a particularly heavyweight dependency. It requires libclang to be available and also adds a considerable amount of time to the compile. If it makes compiling the software easier - and end users compiling the software themselves would benefit here as well as distro maintainers - then it would seem like a pragmatic move to me.

This really isn't about bindgen.

Any time a dependency is changed, the package version should be bumped and retested. This is a massive improvement in reproducibility, bisectability and our ability to QA.

Before this model, bisecting breakage that was a result of a change in library 'a' breaking something in consumer 'b' was effectively impossible; now it is.

What Debian and Fedora have been doing is a big step backwards, this whole 'unbundling of dependencies' (that were going to be statically linked anyways) needs to die.

Re: Orphaning bcachefs-tools in Debian

#75

Post by rfoo »

AshamedCaptain wrote: post #17
viraptor wrote: post #5

Earlier quoted context omitted.

This isn't Rust specific. The same issue exists in all languages where the versions can be restricted at project level. It's an issue in Debian because they can't handle multiple concurrent versions (beyond renaming the package) and want every package built without internet access. But that same issue affects Ruby, Python, etc. if the project specifies strict versions. And if you deal with filesystems, you really wan…

In fact, I am tired of the same issue e.g. when evaluating machine learning tools with Python. It's as if everyone just says "dependency = ${exact_version_I_have}" and never considers anything else. Not helped by the fact that most of these projects break ABI every other day which I still think it is just plain evil. Please don't say "just use containers/virtualenvs/vendoring/whatever" because at some point you obvio…

> And your GPU driver is only going to be at one version, no matter how many statically linked executables you have.

This is a non-issue.

libcuda.so (which is part of the NVIDIA driver, their userland part) can not be statically linked into anything, since it's closed-source and no .a shipped at all, so no possibility of misconfigure it to be statically linked. And NVIDIA drivers, as a whole (including a pair of libcuda.so and corresponding .ko), is infinitely backward-compatible, you can always use new driver with old libraries. So the problem you mentioned does not exist at all.

Statically linking to cuBLAS / cuDNN etc never add more pain and can only reduce pain. Unless you hold it wrong.

Re: Orphaning bcachefs-tools in Debian

#76

Post by greatgib »

nubinetwork wrote: post #22
viraptor wrote: post #5

Earlier quoted context omitted.

This isn't Rust specific. The same issue exists in all languages where the versions can be restricted at project level. It's an issue in Debian because they can't handle multiple concurrent versions (beyond renaming the package) and want every package built without internet access. But that same issue affects Ruby, Python, etc. if the project specifies strict versions. And if you deal with filesystems, you really wan…

> This isn't Rust specific. The same issue exists in all languages where the versions can be restricted at project level Can confirm, I occasionally use an ai program that has to hard code all of their python dependencies, because it's the only way to get it to compile, let alone run properly... and then they go and change the underlying package manager, and figure out they have to hard code even more... it's a blood…

I think that 2 different things are mixed up.

For Rust it is language specific, because the thing being statically built, doing differently is very hard.

For other languages like Python, there is not a need and you should not do that. The dynamic part of it make it so that developer should be resilient to different versions. And python also provide a lot of conveniences for that (just think about six).

But in your case, you faced the current situation of lot of "data scientist" jumping the band wagon of AI are creating a lot of things like apprentice sorcerers without a real experience of software engineering.

Re: Orphaning bcachefs-tools in Debian

#77

Post by AshamedCaptain »

rfoo wrote: post #75
AshamedCaptain wrote: post #17

Earlier quoted context omitted.

In fact, I am tired of the same issue e.g. when evaluating machine learning tools with Python. It's as if everyone just says "dependency = ${exact_version_I_have}" and never considers anything else. Not helped by the fact that most of these projects break ABI every other day which I still think it is just plain evil. Please don't say "just use containers/virtualenvs/vendoring/whatever" because at some point you obvio…

> And your GPU driver is only going to be at one version, no matter how many statically linked executables you have. This is a non-issue. libcuda.so (which is part of the NVIDIA driver, their userland part) can not be statically linked into anything, since it's closed-source and no .a shipped at all, so no possibility of misconfigure it to be statically linked. And NVIDIA drivers, as a whole (including a pair of libc…

That is hardly true for NVIDIA over the years, as I have experienced personally, and it's definitely not true for about every other GPU driver in existence -- particularly AMD's RoCM, which even recommends specific Python versions to go with each ROCM version. In addition, replace "GPU" with any other piece of hardware, such as NPU. Or even software, such as your desktop environment. Do you think a statically linked binary will run with Wayland+1 ? It's already a chore to run games that link with older SDL versions, and the only reason you can at all is that SDL was dynamically linked...

Even the kernel itself does a pretty poor job at maintaining backwards compatibility (e.g. contents of /sys changing every other year). Statically linking just hardcodes one of these things, and then forgets about the rest, such as IPC APIs, desktop APIs, filesystem structure, etc. which due to the poor culture out there are moving as much as the libraries themselves.

Re: Orphaning bcachefs-tools in Debian

#78

Post by sp1rit »

frankjr wrote: post #28

The list of dependencies doesn't really seem crazy. Logging, parsing command line arguments, band-aiding error traits, working with uuids, reading binary data, "memset_s". All of those should be a part of the standard library as in any other programming language. The fact that you simply cannot build anything but trivial examples without taking a dependency on a third party library is just plain ridiculous and hinder…

Still, if you include transitive dependencies you end up with a total of 90 dependencies[0] which is unheard of in systems programming. This for some reason includes stuff like "winapi-i686-pc-windows-gnu" which really has no place for a set of linux software tools.

[0]:https://github.com/gentoo/gentoo/blob/master/sys-fs/bcachefs...

Re: Orphaning bcachefs-tools in Debian

#79

Post by jt2190 »

peppermint_gum wrote: post #30

>not even considering some hostile emails that I recently received from the upstream developer or his public rants on lkml and reddit It feels like whenever the author of bcachefs comes up, it's always because of some drama. Just the other day he clashed with Linus Torvalds: https://lore.kernel.org/lkml/CAHk-=wj1Oo9-g-yuwWuHQZU8v=VAsB... My reading is that he's very passionate, so he wants to "move fast and break thi…

Is bcachefs-tools going into the mainline distros or into something that’s meant to be less stable and experimental? Linus makes it sound like there’s a more appropriate place for this work.

Edit: Reading through the thread, it seems like there is a claim of rigorous but effectively private testing. Without the ability to audit those results easily it’s causing a lot of worry.

Re: Orphaning bcachefs-tools in Debian

#80

Post by ralferoo »

peppermint_gum wrote: post #30

>not even considering some hostile emails that I recently received from the upstream developer or his public rants on lkml and reddit It feels like whenever the author of bcachefs comes up, it's always because of some drama. Just the other day he clashed with Linus Torvalds: https://lore.kernel.org/lkml/CAHk-=wj1Oo9-g-yuwWuHQZU8v=VAsB... My reading is that he's very passionate, so he wants to "move fast and break thi…

It's very clear from that thread that he doesn't understand the purpose of the stable branch. It doesn't mean "stable" as in "the best possible experience", it means it as in "this code has been tested for a long period of time with no serious defects found" so that when the stable branch is promoted to release, everything has undergone a long testing period by a broad user base.

If there is a defect found, the change to a stable branch should literally be the minimal code change to fix the reported issue. Ideally, if it's a newly introduced issue (i.e. since being on the stable branch), the problematic code reverted and a different fix to the original defect applied instead (or left if it's deemed less of an issue than taking another speculative fix). Anything that requires a re-organisation of code, by definition, isn't a minimal fix. Maybe it's the correct long-term solution, but that can be done on the unstable branch, but for the stable branch, the best fix is the simplest work around. If there isn't a simple work around, the best fix is to revert everything back to the previous stable version and keep iterating on the unstable branch.

The guy even admits it as well with his repeated "please don't actually use this in production" style messages - it's hard to give a greater indication than this that the code isn't yet ready for stable.

I can understand why from his perspective he wants his changes in the hands of users as soon as possible - it's something he's poured his heart and soul and he strongly believes it will improve his users' experience. It's also the case that he is happy running the very latest and probably has more confidence in it that an older version. The rational choice from his perspective is to always use the latest code. But, discounting the extremely unlikely situation that his code is entirely bug free, that just means he hasn't yet found the next serious bug. If a big code change is rushed out into the stable branch, it just increases the likelihood that any serious bug won't have the time it needs in testing to have the confidence that's the branch is suitable for promotion to release.

Post reply on HN