The modern packager’s security nightmare • Hacker News Board

Re: The modern packager’s security nightmare

#171

Post by lifthrasiir » Mon, Feb 22, 2021, 2:17 AM UTC

bscphil wrote: post #86

hctaw wrote: post #74

Earlier quoted context omitted.

I'd disagree that the problem of large organizations are different from the problems of the FOSS ecosystem. Organizations just have a financial incentive to fix them, the FOSS ecosystem does not. If mutually incompatible dependencies and security updates breaking software weren't problems for both corporate and FOSS ecoystems, these new technologies wouldn't have needed to exist. They'd just use the existing platform…

> They should be cut out of the loop. I disagree. This is how you get the Google Play store or the "freeware" app marketplace. It sucks. As a user, I'm quite happy to continue using a traditional distribution even if it means I don't get to use a handful of flashy programs by developers that disagree with the concept of maintainers. So far that choice has been much more positive than negative for me, and I'm doing my…

> As a user, I'm quite happy to continue using a traditional distribution even if it means I don't get to use a handful of flashy programs by developers that disagree with the concept of maintainers.

As a user, I'm afraid that you don't seem to be a representative of typical users. I would be happy to use a traditional distribution if they don't break on updates, which is still not the case. It's clear that traditional distros were not satisfactory on even keeping their own promises. I know they work hard, but ultimately the visible outcome says everything.

Re: The modern packager’s security nightmare

#172

Post by fomine3 » Mon, Feb 22, 2021, 2:30 AM UTC

nwallin wrote: post #122

paultopia wrote: post #66

I feel like this confuses a lot of things by assuming an extremely sophisticated end user. Sure, if you're a double-threat dev/sysadmin using linux, then when some vulnerability gets discovered in some dynamically linked library on your system, you have the capacity to (a) receive information about that fact, and (b) update it. But now suppose you're an ordinary person. You use software. Maybe you even have a windows…

> (b) You have to either learn what a DLL is and learn how to update it ... That's not at all what the process is for the ordinary person. The ordinary person sees a notification pop-up from "Ubuntu Software Center" that says 8 packages or whatever need to updated, with one button that says "update everything now" or whatever and one that says "remind me later" or whatever. It's up to you to choose a distro that appl…

It's happy that you only use system's Python package. You need manual version management if you install multiple Pythons.

Re: The modern packager’s security nightmare

#173

Post by SAI_Peregrinus » Mon, Feb 22, 2021, 2:36 AM UTC

staticassertion wrote: post #80

dataflow wrote: post #33

Why do only *nix folks pull their hair out about this? On Windows programs bundle their dependencies all the time (sometimes even as shared libraries! but without the independent update benefits) and hardly anybody loses sleep over it. Heck, users actually like the fact that it minimizes friction. Nobody claims it's rock-solid security, but does it need to be? Actually, now that I wrote it above, I think I might have…

> while of course infosec experts will freak out about any vulnerability existing anywhere for even a shred of a nanosecond, in the real world this is really only a big deal for servers, not clients I don't think that the linked article really reflects a consensus among the security community at all. I don't really find the "dynamic libraries are more secure" argument overly compelling, certainly not stated so broadl…

Dynamic libraries can even introduce new vulnerabilities to everything using them.

Re: The modern packager’s security nightmare

#174

Post by orf » Mon, Feb 22, 2021, 2:53 AM UTC

cycloptic wrote: post #163

forrestthewoods wrote: post #148

Earlier quoted context omitted.

I’m not sure I made that comparison. I’m saying the popularity of containers is an existence proof that the distro-oriented shared library system is failing to meet the needs of common use cases. Programs that bundle dependencies have a radically reduced need for snapshots, VMs, or containers. Such tools do provide a variety of value. But they don’t become virtually requirements to merely launch a program without err…

You can't get rid of the distro-oriented shared library system, it exists as long as you are building on top of an operating system. To that end, when you say "we depend on operating system minimum version X.Y.Z" that now becomes another dependency you have to ship at some point if you're running the production machines. Containers are popular because they actually solve that exact problem. If there was some other co…

Parent is saying that the distribution model of modern distros is so legacy and out of alignment with what people actually want that a solution involving bundling the entire damn OS to sidestep that pit of snakes has now became the defacto way to deploy software.

To put it another way, what do you think the relative popularity of distributing your own internal software via a private apt repo is compared to bundling it as a stateless container and putting it on a registry? Some big companies that pre-date containers do it with apt. Most don’t. For good reason.

Re: The modern packager’s security nightmare

#175

Post by pdonis » Mon, Feb 22, 2021, 2:55 AM UTC

hctaw wrote: post #42

bscphil wrote: post #13

Over the last several weeks I was working on an essay about this exact problem, including the connection between static linking and bundling. This one is so well done that I probably won't even publish it. But I'll add this, for people who may not immediately see why this is important. I think that the real danger these new technologies represent is not inherently bad technology, but the possibility of ecosystem dama…

> The distribution / maintainer / package manager approach has proven to be an extremely reliable way to get trustworthy software. Many of us love it and want to see it stick around. I disagree, it's proven to be inadequate for modern software development and that's why these new languages/ecosystems are springing up. The least reliable way to package and distribute software is by relying on traditional package manag…

>I disagree, it's proven to be inadequate for modern software development

You're not disgreeing with the post you responded to; you're just stating a different priority.

hctaw said the distro/maintainer/pm approach is extremely reliable at producing trustworthy software. That means trustworthy to the user. It says nothing at all about how hard producing that software is for the developer.

You are saying that the distro/maintainer/pm approach makes it harder for the developer. That's true. But it doesn't contradict the above at all.

And anyway, as a user, I don't care. I want my software to work, to be stable, and to not have security flaws; and if a security flaw is found, I want a fix to be pushed to me ASAP. The distro/maintainer/pm approach does that. If instead I have umpteen zillion different statically linked applications installed, each of which packages all of its own dependencies, then instead of just relying on my distro to push security fixes to shared libraries that everyone uses, I have to rely on every single one of those developers to do it for their own packages. And most of them won't do it, or they'll do it when they get around to it instead of when I, the user, need it.

> The universe I'd like to live in is where the only use case for dynamic linking are OS vendor APIs and cryptographically secure functions like TLS

This won't work either, because those are certainly not the only places where security flaws can happen that I, the user, need a fix for ASAP.

Re: The modern packager’s security nightmare

#176

Post by zrm » Mon, Feb 22, 2021, 2:56 AM UTC

rictic wrote: post #126

baybal2 wrote: post #115

Earlier quoted context omitted.

> it's easier to reason about code that is running if you can look at the commit that a bin was built from and know exactly what's inside (including all deps). Believe me, it's usually the opposite. Lack of proper releases, testing, and versioning results in unending checkout-fu to figure out what commits for each of 20 libraries will work for each other. The idea is plainly stupid, without any redeeming qualities. T…

Are you actually talking about static linking here, or just venting your spleen about sloppiness in software engineering practice more generally? Because tracking dependencies in source control (specifically, checking in lock files) is tremendous for reproducibility. It means that when bisecting through the commit history to find when a problem began is not just bisecting through the local source code, but also the s…

> you're able to find the first commit that introduced the issue in O(log(commits)) time, rather than needing O(commits * (num dependencies * dependency versions)) time.

Try the oldest and newest compatible version of your code and the oldest and newest compatible version of each dependency. This is O(num dependencies), which is generally small N and you can often guess which to try first. Now do each version of the code or dependency that actually made a difference. O(log(N)) on the versions of that code. You're at O(num dependencies) + O(log(N)), not O(commits * (num dependencies * dependency versions)).

Also, doing it the other way can often lead to frustration. The actual bug is introduced in commit #1234 but is timing dependent, then a dependency is upgraded from version 2 to version 3 in commit #5678 which tickles the bug, e.g. the new dependency version moved around some cache lines. Now you're looking in entirely the wrong place at an innocent dependency.

Whereas if you notice that the bug exists with dependency version 3 and the latest commit and then do binary search on all the commits while holding the dependency at version 3, plausibly the bug shows up between commit #1233 and #1234.

Re: The modern packager’s security nightmare

#177

Post by 0xbadcafebee » Mon, Feb 22, 2021, 2:57 AM UTC

hctaw wrote: post #42

bscphil wrote: post #13

Over the last several weeks I was working on an essay about this exact problem, including the connection between static linking and bundling. This one is so well done that I probably won't even publish it. But I'll add this, for people who may not immediately see why this is important. I think that the real danger these new technologies represent is not inherently bad technology, but the possibility of ecosystem dama…

> The distribution / maintainer / package manager approach has proven to be an extremely reliable way to get trustworthy software. Many of us love it and want to see it stick around. I disagree, it's proven to be inadequate for modern software development and that's why these new languages/ecosystems are springing up. The least reliable way to package and distribute software is by relying on traditional package manag…

> it's proven to be inadequate for modern software development

Sure, because it's not meant for software development. It's meant for users to run software.

Software development packagaing/distribution is pretty bad. Most languages do it their own way and never seem to learn the lessons of previous ones. And on top of that, they encourage poor habits, like writing your own module that's the same as somebody else's with one new function, rather than contributing to the existing module, or writing an extension for it.

> I do not trust system provided libraries to function with my applications because I've been burned so many times in the past.

Probably this is because both you + the library developer are not coordinating with the distributions on how you release your code. Distros get a lot of flack for breaking changes, but they're working from software released by developers, and rely entirely on their own user base to test changes. If developers cared about their software working they'd be more involved in its packaging & distribution.

Re: The modern packager’s security nightmare

#178

Post by orf » Mon, Feb 22, 2021, 2:59 AM UTC

syllogism wrote: post #20

This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…

You’re completely wrong and this advice is somewhat harmful. What you’re describing is how a Python application should be managed. Not a library. Libraries should absolutely not lock their advertised dependencies to arbitrary point-in-time versions for fairly obvious reasons.

Picking a suitable dependency specifier depends heavily on the maturity of the library you’re using and if you need any specific features added or removed in a specific release.

Saying your library depends on “spacy==2.3.5” is a lie that will mean any other library that depends on spacy>=2.3.6 can’t be used. Even if your code will realistically work fine with any spacy 2.x release.

Re: The modern packager’s security nightmare

#179

Post by syllogism » Mon, Feb 22, 2021, 3:35 AM UTC

dbaupp wrote: post #46

syllogism wrote: post #20

This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…

Libraries pinning dependencies only fixes a narrow portion of the problem and introduces a bunch of others (particularly in ecosystems where only a single version of a package can exist in a dependency tree). In particular, it is great because it makes life slightly easier for the library developers. However, if every library pinned deps, it becomes much harder to use multiple libraries together: suppose an app used…

I think you're really under-rating how important it is to be able to do something like "pip install 'requests==1.0.5'" or whatever, in order to reconstruct the past state of a project. If requests hasn't pinned its dependencies, that command will simply not work. The only way you'll be able to install that version of requests is to manually go back and piece together the whole dependency snapshot at that point in time.

There's pretty much no point in setuptools automatically installing library dependencies for you if you expect the library dependencies to be unpinned. In fact it would be actively harmful --- it just leads people to rely on a workflow that works today but will break tomorrow.

You're asking for an ecosystem where there's no easy way to go back and install a particular version of a particular library. That's not better than having version conflicts.

The other thing I'd note is that it's quite an understatement to say that pinning dependencies makes life "slightly easier" for library developers. We're not going to accept builds just breaking overnight, and libraries that depend on us aren't going to accept us breaking their builds either.

Re: The modern packager’s security nightmare

#180

Post by 1MachineElf » Mon, Feb 22, 2021, 3:39 AM UTC

orra wrote: post #77

hctaw wrote: post #50

The day I stop static linking is the day I can compile on one distro and ship to many without worrying about users reporting loader errors. That day is not today. Until then, I'll keep doing it, because it saves me time and money. I don't really buy that dynamic linking all the things is such a boon to security. But I'll link to this: https://drewdevault.com/dynlib > Not including libc, the only libraries which had "…

> I don't really buy that dynamic linking all the things is such a boon to security. Agreed. It's really not a panacea. When you upgrade a library, you probably want to restart the running applications that depend upon this. Dynamic linking won't save you. The solution is having a graph of your dependencies! IIRC, NixOS gets this right. I don't think Debian's apt did?

I would really like to know how NixOS solves this problem for things like yarn, webpack, parcel, esbuild, npm, pip, conda, cargo, stack, go modules, ruby gems, etc.