Live data from Hacker News

The modern packager’s security nightmare

blogs.gentoo.org

151–160 of 282 posts

Re: The modern packager’s security nightmare

#151

Earlier quoted context omitted.

Honest question: have you ever worked as an application developer? Responsible for getting working artifacts to users as a means to an end? Pinning dependencies absolutely and unquestionably works better, and for longer, than dynamic linking, for this use case.

Perhaps I still fail to explain myself: what I am saying is that _not pinning_ only _adds_ more choices, so by definition it can only work better. Pinned or not, if a software update breaks things, you can always just revert back to a previous version of your dependencies. This applies to a myriad soft problems including a dependency changing interface. However, when pinning, when one of your static dependencies is b…

> However, when pinning, when one of your static dependencies is broken due to a change outside your control (e.g. hardware, operating system, security issue making it unusable, or something else), the user's only recourse is to call the developer to fix the software.

In practice, this happens so infrequently it can be ignored as a risk. (When it does happen, users generally don't expect the software to continue to work.)

> dynamic linking has at least one extra advantage...

You don't seem to be acknowledging the downside risk to dynamic linking which motivates the discussion in the first place. An update to a dynamically linked dependency which breaks my delivered artifact is an extremely common event in practice.

Re: The modern packager’s security nightmare

#152
post #124

Earlier quoted context omitted.

> it's easier to reason about code that is running if you can look at the commit that a bin was built from and know exactly what's inside (including all deps). Believe me, it's usually the opposite. Lack of proper releases, testing, and versioning results in unending checkout-fu to figure out what commits for each of 20 libraries will work for each other. The idea is plainly stupid, without any redeeming qualities. T…

> Lack of proper releases, testing, and versioning results in unending checkout-fu to figure out what commits for each of 20 libraries will work for each other. Admittedly I'm basing this on my experience in a very large monorepo environment, but there's no figuring out which commits will work with each other. Every commit with every library will work, otherwise it doesn't get committed. Yes, this involves massive CI…

Distros are great for off-the-shelf software especially if you don't care which version you get too much. When versions matter, you quickly get into dependency-hell. So long-lived software tend to stabilize, and then remain unchanged.

K8s, go and even ruby, tend to change and evolve. It's usually a bad idea to pull such software from distros then, even if available.

The means to get software is simply too different, and it's a non-problem for everyone but completist distro maintainers.

Re: The modern packager’s security nightmare

#153
post #149

Earlier quoted context omitted.

This makes me wonder if there's a Linux base system suitable for servers that embraces the newer approach. That is, a minimal base system that's built for immutable container images, providing just what's needed to bootstrap the current generation of language-specific build and package systems. The Alpine Linux Docker images might be a good choice for now, but IIUC, Alpine Linux itself still embraces the older distro…

> That is, a minimal base system that's built for immutable container images, providing just what's needed to bootstrap the current generation of language-specific build and package systems. At least for the first half, that sounds sort of like Fedora/Red Hat CoreOS[0], the predecessor CoreOS fork Flatcar Container Linux[1], or the Amazon distribution BottleRocket[2]. [0] https://getfedora.org/en/coreos?stream=stable…

Fun thing about BottleRocket relevant to this thread: it uses Cargo as its build system. It's really wild and very interesting and more people should know about it, IMHO.

Re: The modern packager’s security nightmare

#154

Correct me if I'm wrong (really) but isn't static linking mostly a problem for packagers? The less a package maintainer modifies the application the better IMO. If application A and B rely on dependency D, which turns out to have a vulnerability, fixed in D', Then _why_ do we think it is anyone but A & B's developers responsibility to update to D' and distribute patched versions? If the packager tries to do it, the c…

> Correct me if I'm wrong (really) but isn't static linking mostly a problem for packagers? The less a package maintainer modifies the application the better IMO.

IMHO that's precisely why dependencies should be unpinned.

Let's say application A relies on dependencies B, C, and D, and dependencies B, C, and D depend on dependency E. Let's say dependency E has a critical security vulnerability and needs to be updated today to E'.

Let's say you have unpinned versions:

The packager for E updates E to E'. The end.

Let's say you have pinned versions:

The developer for B updates package B to depend on E', the developer for C updates C to depend on E', the developer for D updates D to depend on E', the developer for A updates A to depend on B', C', and D'. The packager for B updates the package for B, the packager for C updates the package for C, the packager for D updates the package for D, and the packager for A updates the package for A.

You'll notice that there's a timing issue here. The packager for C cannot move until the developer for C has done the work, the developer for A cannot move until the developer for B, C, and D have done their work, and the packager for A cannot do anything until everyone has completed their work. If, for instance, the developers for C all live in Texas and their power's been out for a few days, and when they get power back they're busy with other stuff for a while, it might take quite some time for C's developers to get an official package posted. But it's that important that A gets updated, because A is a network service with a port open to the internet and E is openssl or whatever. So now what?

In a perfect world, all software dependencies would have active, attentive, prompt maintainers, but it tends to not be that way. Lots of critical internet infrastructure packages have a maintainer who's just some random person in Nebraska, and they go on vacation, or lose interest, go to sleep at night, go to little league games on the weekend, some of them have day jobs. If we lived in a world where Apache can't be updated to use the latest dynamic library for openssl because the developer for leftpad is watching a movie and has their phone turned off, that's a very serious problem, and it's a crazy world I would not want to be a sysadmin in.

Certainly, maybe the packager for E is gonna be off this week, but a distro's packaging team tends to have a much easier time filling in for a maintainer who's away if the package is loosely coupled with the application's build process. 90% of the time, if a package in Gentoo requires an update, all you need to do is `mv foo-1.2.3.ebuild foo-1.2.4.ebuild`, `repoman manifest` and git commit+push. (I can't speak for other distros.)

The system isn't perfect, but IMHO it's much more robust to the unfortunate realities of the ugly, soft underbelly of the world than static linking is.

Re: The modern packager’s security nightmare

#155
post #44

Earlier quoted context omitted.

You mean cgroups, or zones don’t you? Docker (was, last time I heard) a security disaster, not generating robust layer hashes, lacking user isolation, and plenty just running as root...

There's more to containers on Linux than just Docker.

To be fair you need to go to the hypervisor level (like Firecracker) to get any decent level of sandboxing.

Re: The modern packager’s security nightmare

#156

This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…

[deleted]

Re: The modern packager’s security nightmare

#157

Here is the solution to this Debian/Gentoo/ packager's dilemma. Packagers realize that their numbers are small and they can't keep up fixing all the modern big software (BS) projects that go against their philosophy. They define the core of the OS that they can keep managing in this traditional way (kernel, basic userland, basic desktop, basic server services, basic libraries, security support for all that). "But peo…

That's literally what windows does, for the most part, actually.

Re: The modern packager’s security nightmare

#158

Yeah well, who's hungry? No really, neither users nor developers care, nor should they. I've been using Linux on the desktop for well over a decade and I'm tired of seeing this plea for everything to behave exactly like C or scripting languages because every distro wants to be its own special snowflake and it would be too hard to adapt. The world has changed and distros have to stop pretending it hasn't. Flatpak, Nix…

It's hard not to read this and not feel like a little bit of it is some maintainers worried about losing control. When I get on irc and say the words "pip" they certainly send me those vibes.

Re: The modern packager’s security nightmare

#159

Earlier quoted context omitted.

The most frustrating thing is that pip doesn't make it easy to use more loose declared dependencies while freezing to actual concrete dependencies for deployment. Everybody rolls their own. > Python libraries should not pin dependencies. _Applications_ can pin dependencies, including all recursive dependencies of their libraries. Is the pypi package awscli an application or a library? poetry is frustrating in that it…

> Is the pypi package awscli an application or a library? Hopefully a library! As hopefully the AWS command-line interface is maintained and distributed separately from any SDK that powers it...

boto core/boto/boto 3 are the libraries, which awscli drives.

Re: The modern packager’s security nightmare

#160

Earlier quoted context omitted.

Perhaps I still fail to explain myself: what I am saying is that _not pinning_ only _adds_ more choices, so by definition it can only work better. Pinned or not, if a software update breaks things, you can always just revert back to a previous version of your dependencies. This applies to a myriad soft problems including a dependency changing interface. However, when pinning, when one of your static dependencies is b…

> However, when pinning, when one of your static dependencies is broken due to a change outside your control (e.g. hardware, operating system, security issue making it unusable, or something else), the user's only recourse is to call the developer to fix the software. In practice, this happens so infrequently it can be ignored as a risk. (When it does happen, users generally don't expect the software to continue to w…

> In practice, this happens so infrequently it can be ignored as a risk.

Well I disagree there. Security issues or external protocol changes (e.g. TLSv1.2 to TLSv1.3) are rather frequent, not to mention usually customer wants to upgrade their machines (old ones broke) and existing operating system no longer supports the new hardware.

> An update to a dynamically linked dependency which breaks my delivered artifact is an extremely common event in practice.

Again, I agree. A "surreptitious" dependency update breaking the software is much more common. However, I have already acknowledged that _two times already_, and the point that I'm making is that it doesn't matter if you are pinning dependencies or not: customer CAN FIX these issues without help from the developer. They just have to roll back the update!

On the other hand customer CAN'T fix the first issue (e.g. new hardware).

Post reply on HN