Live data from Hacker News

The modern packager’s security nightmare

blogs.gentoo.org

11–20 of 282 posts

Re: The modern packager’s security nightmare

#11
post #8

Correct me if I'm wrong (really) but isn't static linking mostly a problem for packagers? The less a package maintainer modifies the application the better IMO. If application A and B rely on dependency D, which turns out to have a vulnerability, fixed in D', Then _why_ do we think it is anyone but A & B's developers responsibility to update to D' and distribute patched versions? If the packager tries to do it, the c…

The OP is talking about technical details like static linking and language ecosystems, but if you zoom one level out, this comment correctly pins where the underlying problem actually lies: that the distribution maintainers inject themselves into the development process of all the software they ship. When A/B are not well-maintained upstream, distributions can assist by updating its dependency D, but even when A/B ar…

I think it would help if distributions could snatch a piece of real estate in upstream software. Something like, "in every project, the debian/ root folder belongs to the debian project and follows their rules". The packaging could then verify this folder and put their patches, build scripts, etc. there. This would help upstream communication a lot, I guess.

Re: The modern packager’s security nightmare

#12
Yeah well, who's hungry?

No really, neither users nor developers care, nor should they. I've been using Linux on the desktop for well over a decade and I'm tired of seeing this plea for everything to behave exactly like C or scripting languages because every distro wants to be its own special snowflake and it would be too hard to adapt.

The world has changed and distros have to stop pretending it hasn't. Flatpak, Nix, those are better models that reflect our reality where developers don't want to worry about ten thousand different distros with wildly different downstream configurations producing different kinds of bugs, as well as the desire of users of just getting the darned app.

If you're worried about security you must always, first, work with upstream. Upstream should be the party responsible for its users, not you. Upstream not fast enough and you want to go the extra mile? Well then your packaging tool-belt should have support for patching a series of libaries in an automated fashion and rebuilding the dependency tree; and make sure that you return to upstream as soon as possible.

If you want your downstream to diverge from upstream because you want to act as a barrier as the old distros do, then you'll have to accept the fact that you're maintaining a fork and not pretend to upstream that you're distributing their work or that your bug reports are compatible. Otherwise, again, just limit yourself on distributing the latest upstream's stable release with the bare minimum patching necessary for getting things to work with the distro's chosen config.

With some luck, after the Linux community comes to terms with the fact that the distro model must shift, we can begin to finally share some packaging efforts between all distros and we can leave much of the incompatibility bullshit in the past.

Imagine one day everyone just building from shared Nix derivations? Very ironically, it would look a lot like Portage and Gentoo's USE flags but with everyone building off those generic derivations and offering binary caches for them.

Re: The modern packager’s security nightmare

#13
Over the last several weeks I was working on an essay about this exact problem, including the connection between static linking and bundling. This one is so well done that I probably won't even publish it.

But I'll add this, for people who may not immediately see why this is important. I think that the real danger these new technologies represent is not inherently bad technology, but the possibility of ecosystem damage. The distribution / maintainer / package manager approach has proven to be an extremely reliable way to get trustworthy software. Many of us love it and want to see it stick around. And it's been possible because "upstream" developers in the open source ecosystem have been willing (or forced) to work with distributions to include their software. But this seems to be changing. A highlight from my essay:

'Many software projects are not good citizens in the open source ecosystem. They are working with a model of development and distribution that does not mesh well with how open source software is predominantly produced. ... These [new] languages are now developing their own ecosystems, with their own expectations for how software should be created and distributed, how dependencies should be handled, and what a "good piece of software" looks like. Regardless, an increasing amount of software is being built in these ecosystems, including open source software.

There might come a day in which open source is fractured. Two different communities create two very different kinds of software, which run on the same systems, but are created and distributed in very different ways. I begin to worry that the community I care about might not survive. Even if my ecosystem continues on its way, I don't want this split to take place. I think being part of the open source ecosystem is good for software, and I think having all the software you could want available within that ecosystem is good for users. If anything, this essay is a call to those who agree to be more careful about the software they create. Make sure it's something that Debian maintainers could be proud to ship. If you're working on a Rust program, for example, ask questions like "how can I make this program as easy for maintainers to distribute as possible?" If you have the ability to work on projects like Rust or Go, do what you can to give their applications the ability to easily split dependencies and support system provided libraries. Let's try to make sure the software ecosystem we love is around for the next generation.'

I'd also strongly recommend the essay "Maintainers Matter" for another take on why this is so important. http://kmkeen.com/maintainers-matter/ Or read about Debian's recent problems with vendoring: https://lwn.net/Articles/842319/

Re: The modern packager’s security nightmare

#14
Would it be feasible for rust to ship "re-link" scripts? That is, when a cargo dependency gets updated, the distribution can check whether packages code needs to be re-linked? Similarly, when distributed code (say libssl) changes, the rust packages can be re-linked by the distribution on-site?

Re: The modern packager’s security nightmare

#15
post #9

Correct me if I'm wrong (really) but isn't static linking mostly a problem for packagers? The less a package maintainer modifies the application the better IMO. If application A and B rely on dependency D, which turns out to have a vulnerability, fixed in D', Then _why_ do we think it is anyone but A & B's developers responsibility to update to D' and distribute patched versions? If the packager tries to do it, the c…

The distribution provides support for much longer than upstream does. Also, packaging is actual work. Just because some hipsters in some company consider it cool to release new features every 4 weeks, it does not mean that the volunteers of some linux distribution can keep up with thag. So if you want your distribution to work well, you should focus on workable and transparent interfaces. If you are happy with the su…

> on the other hand you don't need a distribution in the first place.

I think you're right about that, I personally want as little distribution as possible. FreeBSD ports or Arch AUR work well for me.

The idea of solidifying other people's software into a bundle and then maintaining it (with necessarily limited expertise) seems like a loosing battle.

Re: The modern packager’s security nightmare

#16
post #5

>> Why do people pin dependencies? The primary reason is that they don’t want dependency updates to suddenly break their packages for end users, or to have their CI results suddenly broken by third-party changes. Or because we dont want accidental or malicious security vulnerabilities to get automatically incorporated into the software. This stuff works both ways. You dont automatically incorporate fixes, nor new pro…

The vast, vast majority of updates fix security issues. It's like not vaccinating in case you're one of the million people that has an allergic reaction. Supply chain attacks are rare, not the norm. We hear about such things (and only rarely at that) because it's exceptional enough to make the news.

Re: The modern packager’s security nightmare

#17
post #7

Fantastic article. I now have something to point people to when they ask "what is wrong with pinning?" or "what is wrong with static linking?" or "why can't you just use pip?" Michał has had to deal with some pretty crazy stuff the last couple of months ... scratch that, years. The recent attempts to empower developers to distribute their own software means that there is now the potential for there to be as many bad…

> "why can't you just use pip?"

IMO, pip is great, but it has exactly two use cases where it's warranted. One is where you need to be your own maintainer, e.g. you've developed a private web server to run your website, and you need to manage its dependency versions precisely. The other is for development purposes: it's really great to be able to use different versions of Python or test against different libraries than your distribution ships.

It's not (or shouldn't be) for shipping software to end users. That's the problem being complained about in the essay. This particular way of handling dependencies (use system libraries by default, but allow the user / developer to create an entire virtualized Python installation if they want one) is actually why I think Python handles this better than most other languages, including older ones.

Re: The modern packager’s security nightmare

#18
This article doesn't make a good case against static linking, and the author doesn't seem to understand what vendoring is either:

> Bundling (often called vendoring in newspeak) means including the dependencies of your program along with it.

No, vendoring means including a copy of the source code of dependencies in your repo. You can bundle dependencies without vendoring them.

The only argument presented against static linking is that when a library is updated rebuilding dependants takes longer (and people will have to download bigger updates but I doubt many people care about that).

That may be true, but is it really that big of an issue? I somewhat doubt it (for sane distros that don't make users build everything from source themselves anyway).

The author clearly doesn't like how people do development these days but hasn't stopped to think why they do it like that.

Re: The modern packager’s security nightmare

#19
Here is the solution to this Debian/Gentoo/ packager's dilemma. Packagers realize that their numbers are small and they can't keep up fixing all the modern big software (BS) projects that go against their philosophy.

They define the core of the OS that they can keep managing in this traditional way (kernel, basic userland, basic desktop, basic server services, basic libraries, security support for all that).

"But people want Firefox, Rust, Kubernetes, modern OS has to provide them".

No it doesn't. Let the new complicated juggernaut software be deployed and configured by the users, according to developers' instructions, using whatever modern mechanism they prefer, containers/chroots/light-weight VMs, whatever. Packagers can then forget about the nasty big software and focus on quality of the core OS. Users will have all the newest versions of BS they need from the developers directly.

There is no reason to keep fast-pace, ever-changing and very complicated programs/projects in the main OS distribution. It only leads to lots of OS-specific packaging work with mediocre and obsolete results, and that leads to discontent users, developers and packagers.

Re: The modern packager’s security nightmare

#20
This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating.

If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no longer work. The user will have to then go to significant trouble to find the set of versions that worked at the time.

In short unpinned dependencies mean hopeless bit-rot. It guarantees that your system is a fleeting thing; that you will be unable to today publish an end-to-end set of commands that will work in 2025. This is completely intolerable for practical engineering. In order to fix bugs you may need to go back to prior states of a system and check behaviours. If you can't ever go back and load up a previous version, you'll get into some extremely difficult problems.

Of course the people who are doing the work to actually develop these programs refuse to agree to this. No we will not fucking unpin our dependencies. Yes we will tell you to get lost if you ask us to. If you try to do it yourself, I guess we can't stop you, but no we won't volunteer our help.

It's maddening to hear people say things like, "Oh if everyone just used semantic versioning this wouldn't be a problem". Of course this cannot work. _Think about it_. There are innumerable ways two pieces of code can be incompatible. You might have a change that alters the time-complexity for niche inputs, making some call time-out that used to succeed. You might introduce a new default keyword argument that throws off a *kwargs. If you call these things "breaking" changes, you will constantly be increasing the major version. But if you increase the major version every release, what's the point of semver! You're not actually conveying any information about whether the changes are "breaking".

Post reply on HN