The modern packager’s security nightmare

That's not at all what the process is for the ordinary person. The ordinary person sees a notification pop-up from "Ubuntu Software Center" that says 8 packages or whatever need to updated, with one button that says "update everything now" or whatever and one that says "remind me later" or whatever.

It's up to you to choose a distro that applies the appropriate amount of rigor with regards to testing dynamic library updates. For bleeding edge distros like Gentoo or Arch, it's not that much. Upstream publishes an upstream, and the Gentoo package maintainer chucks it into the testing branch. After 30 days, if no one complains, it gets marked stable. The user chooses a certain amount of risk. (although it's been several years at least since ABI breakage has been an issue for me on Gentoo testing) (note that security critical updates are fast tracked into stable) For other distros like RHEL and Debian stable, the package maintainer spends considerably more effort ensuring a random update of openssl-1.1.1i to openssl-1.1.1j doesn't break stuff. The user chooses a certain amount of stability at the expense of not having the latest version of whatever.

On the other hand, on my Windows computer at work, the process for updating is significantly more intrusive. Few OS updates can be applied without a reboot, Visual Studio updates do not permit me to continue working during an update, there are half a dozen auto-updaters, and some programs which don't get updated unless I manually go to their website and check for an update. I don't know when the last time I updated 7-zip is?

Within the past week, there was a bug report that Python has an RCE with untrusted floats or something. On my linux systems, the package manager had an update within hours, and because there is only one Python installation on each of my linux machines, I know that all applications leveraging Python are now protected from that RCE. On my Windows work machine, I have not been notified that any of the applications I use which embed Python need to be updated. Presumably, this means my Windows machine is vulnerable.

Admittedly I'm basing this on my experience in a very large monorepo environment, but there's no figuring out which commits will work with each other. Every commit with every library will work, otherwise it doesn't get committed. Yes, this involves massive CI infra and tooling to aid in refactoring.

You want to make a breaking change to a lib? Great, it's on you to update every piece of code that calls it.

It's great when you can control every piece of your infra, but I totally get how it's unfeasible (and maintainer hell) for the distro community.

Because tracking dependencies in source control (specifically, checking in lock files) is tremendous for reproducibility. It means that when bisecting through the commit history to find when a problem began is not just bisecting through the local source code, but also the specific versions of every dependency.

So regardless of whether the issue you're investigating is in the package, a dependency, or an unexpected interaction between the two, you're able to find the first commit that introduced the issue in O(log(commits)) time, rather than needing O(commits * (num dependencies * dependency versions)) time.

The underlying issue is that there are no standards for how to package software in a multi language environment. If I want to go from a state where (require 'module-name) fails to a state where (require 'module-name) succeeds, there are a potentially infinite number of ways that that could be accomplished, a single software project cannot ever specify all the possible ways for building software. What they can try to do use uniform interfaces and standard patterns for building their software, in a way that delegates dependency management to an external system. It seems that it is hard for software developers to admit that other people know more about how and where their software will be running than they do.

Good engineering practice seems to dictate that dependency and environment management should be completely orthogonal to the development of an individual component or individual functionality. The op is 3 stories about what happens when the two are not kept orthogonal. The fundamental problem is that it is often easier for individual projects to make decisions that conflate the individual project with its dependencies (no longer orthogonal).

To my knowledge, there is not a universal or well understood set of requirements that could be used to specify what a stable interface between an individual software project and its dependencies looks like. There are a number of candidates, such as gentoo ebuilds, rpm spec files, etc. however I have not seen one that effectively accommodates all of them. Further, there are languages where the implementation (or even design) makes it impossible to keep dependencies and individual projects orthogonal.

The end result of non-orthognal systems is more work for everyone, more wasted cpu cycles, and worse security. Distros can't stop people from using languages that conflate the two, but they can tell them that they are on their own, and that the distros can't depend on components written in such languages in the core of the OS. To everyone pushing the rewrite it in Rust meme this should be a wakeup call. The current design decisions in the language and limitations of the implementation make it less secure than C or C++ because swapping out dependencies is bottlenecked by the centralized primary development team, and maintainers and users can't take orthogonal action to fix an issue.

Every package installed with Nix is isolated into content-addressable* directories, so for example, my install of Firefox is located at /nix/store/c7pmng2x05dkigpbhnjs8fdzd8kk31np-firefox-85.0.2/bin/firefox. This is pretty inconvenient to use directly, so Nix generates a profile that symlinks all your packages into one place (eg. /run/current/system/sw, ~/.nix-profile), and then environment variables like PATH can just include /bin.

With this approach, I can have multiple versions of the same package installed simultaneously, without them conflicting with each other. Like in a traditional distro, any dependencies that are shared between packages aren't duplicated, but if a package needs to explicitly depend on a different version, it can.

Also, because Nix is designed as a functional package manager for building packages from source (even though it has a binary cache), you can trace back exactly what sources were used to build your package and its dependencies, all the way back to the bootstrap binaries used to build any self-hosting compilers (gcc, rust, openjdk, ...)

* Most packages use a hash that's generated from the inputs used to build it, rather than the output that's generated.