The modern packager’s security nightmare • Hacker News Board

Re: The modern packager’s security nightmare

#181

Post by syllogism » Mon, Feb 22, 2021, 3:41 AM UTC

orf wrote: post #178

syllogism wrote: post #20

This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…

You’re completely wrong and this advice is somewhat harmful. What you’re describing is how a Python application should be managed. Not a library. Libraries should absolutely not lock their advertised dependencies to arbitrary point-in-time versions for fairly obvious reasons. Picking a suitable dependency specifier depends heavily on the maturity of the library you’re using and if you need any specific features added…

Everyone needs commands like "pip install 'spacy==2.3.5'" to work reliably in the future, so that you can go back and bisect errors. You need to be able to get back to a particular known-good state, and work through changes

I'm not saying we pin our dependencies to exact specific versions, but we absolutely do set an upper bound, usually to the minor version.

Re: The modern packager’s security nightmare

#182

Post by syllogism » Mon, Feb 22, 2021, 3:43 AM UTC

foerbert wrote: post #43

syllogism wrote: post #20

This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…

I don't have a particularly strong viewpoint on this, but I find it noteworthy that in your example the user themselves is asking for a specific version of the software. You don't seem to be intending for users to ask for simply the latest version and have that work, but a specific one, and you want that specific version to work exactly as it did whenever it was published. I can see some instances in which this expec…

> I'm not sure how to best reconcile these two competing interests.

What would help a lot is if the requirements were specified outside of the actual artifact, as metadata. Then the requirements metadata could be updated separately.

Re: The modern packager’s security nightmare

#183

Post by dbaupp » Mon, Feb 22, 2021, 3:44 AM UTC

syllogism wrote: post #179

dbaupp wrote: post #46

Earlier quoted context omitted.

Libraries pinning dependencies only fixes a narrow portion of the problem and introduces a bunch of others (particularly in ecosystems where only a single version of a package can exist in a dependency tree). In particular, it is great because it makes life slightly easier for the library developers. However, if every library pinned deps, it becomes much harder to use multiple libraries together: suppose an app used…

I think you're really under-rating how important it is to be able to do something like "pip install 'requests==1.0.5'" or whatever, in order to reconstruct the past state of a project. If requests hasn't pinned its dependencies, that command will simply not work. The only way you'll be able to install that version of requests is to manually go back and piece together the whole dependency snapshot at that point in tim…

Sure, it sucks that unpinned dependencies lose historical context as the deps move forward, and I’ve personally suffered this in my own library maintenance work... but there’s still the fundamental issue of conflicting pinned versions if there’s multiple libraries.

(At the app level, the right approach to “going back in time” is for those apps to pin all their deps, with a lockfile or ‘pip freeze’, not just top level ones. That is, one records the deps of requests==1.0.5 in addition to requests itself.)

Re: The modern packager’s security nightmare

#184

Post by tadfisher » Mon, Feb 22, 2021, 3:58 AM UTC

1MachineElf wrote: post #180

orra wrote: post #77

Earlier quoted context omitted.

> I don't really buy that dynamic linking all the things is such a boon to security. Agreed. It's really not a panacea. When you upgrade a library, you probably want to restart the running applications that depend upon this. Dynamic linking won't save you. The solution is having a graph of your dependencies! IIRC, NixOS gets this right. I don't think Debian's apt did?

I would really like to know how NixOS solves this problem for things like yarn, webpack, parcel, esbuild, npm, pip, conda, cargo, stack, go modules, ruby gems, etc.

Usually there are tools to extract the necessary dependency information (url and hash) from a package manager's lockfile and produce a fixed-output derivation so a Nix expression can reproduce the build environment. I know of yarn2nix, cargo2nix, and node2nix, and I wrote and maintain gradle2nix for JVM projects.

Re: The modern packager’s security nightmare

#185

Post by tadfisher » Mon, Feb 22, 2021, 3:59 AM UTC

forrestthewoods wrote: post #101

sigjuice wrote: post #73

Earlier quoted context omitted.

The day I stop static linking is the day I can compile on one distro and ship to many without worrying about users reporting loader errors. That day is not today. Maybe do the trendy thing and link your whole distro together with your app (a.k.a containers)?

The fact that containers exist and are prevalent is a damning indictment on the Linux model imho. Producing software that can simply launch is so wildly complex that the solution is... to snapshot an entire operating system install to run the application. That’s INSANE. But it’s unfortunately required. https://xkcd.com/1987/ > Now, for the worst of all — one that combines all the aforementioned issues, and adds even…

I'll keep beating this drum, but Nix and Guix are the best solution to this problem.

Re: The modern packager’s security nightmare

#186

Post by nwallin » Mon, Feb 22, 2021, 4:17 AM UTC

fomine3 wrote: post #172

nwallin wrote: post #122

Earlier quoted context omitted.

> (b) You have to either learn what a DLL is and learn how to update it ... That's not at all what the process is for the ordinary person. The ordinary person sees a notification pop-up from "Ubuntu Software Center" that says 8 packages or whatever need to updated, with one button that says "update everything now" or whatever and one that says "remind me later" or whatever. It's up to you to choose a distro that appl…

It's happy that you only use system's Python package. You need manual version management if you install multiple Pythons.

You do not need manual version management if you have multiple versions of Python installed. By default, it will use whichever interpreter supported by the application is listed first in PYTHON_TARGETS. If you want, you can override that by calling the python interpreter you want manually, eg `python3.8 ` or `pypy `. But if python3.9 is the "default" interpreter (because it's listed first) but the application only claims support for python3.8 in the ebuild, if you just run the application with no qualifiers it will start the python3.8 interpreter.

Obviously there were many, many years when python2 and python3 needed to be installed side by side, and it's reasonably common for people to have multiple versions of python3 installed side by side. My VPS has both python3.9 and python3.8 installed side by side, for instance, because apparmor is slow to pick up python3.9 support.

Re: The modern packager’s security nightmare

#187

Post by u801e » Mon, Feb 22, 2021, 4:26 AM UTC

orf wrote: post #174

cycloptic wrote: post #163

Earlier quoted context omitted.

You can't get rid of the distro-oriented shared library system, it exists as long as you are building on top of an operating system. To that end, when you say "we depend on operating system minimum version X.Y.Z" that now becomes another dependency you have to ship at some point if you're running the production machines. Containers are popular because they actually solve that exact problem. If there was some other co…

Parent is saying that the distribution model of modern distros is so legacy and out of alignment with what people actually want that a solution involving bundling the entire damn OS to sidestep that pit of snakes has now became the defacto way to deploy software. To put it another way, what do you think the relative popularity of distributing your own internal software via a private apt repo is compared to bundling i…

The company I work at has an internal yum repository that we use for application and certain application dependenices. It's worked reasonably well for us, mainly because we stick with using dependencies provided through the internal or public yum repositories or some other public external yum repositories if a particular dependency is not available otherwise.

I'm sure a similar solution could also work with apt.

Re: The modern packager’s security nightmare

#188

Post by u801e » Mon, Feb 22, 2021, 4:33 AM UTC

anderskaseorg wrote: post #53

syllogism wrote: post #20

This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…

If you publish a Python library with pinned dependencies, your code is broken as soon as someone tries to use it with another Python library with pinned dependencies, unless you happened to pin exactly the same version of the dependencies you have in common. Python libraries should not pin dependencies. _Applications_ can pin dependencies, including all recursive dependencies of their libraries. There are tools like…

> Python libraries should not pin dependencies. _Applications_ can pin dependencies, including all recursive dependencies of their libraries.

This is essentially what we do where I work. When we maked a tagged release, we will create a new virtual environment, run a pip install, run all the tests and then run pip freeze. The output of pip freeze is what we use for the install_requires parameter in the setup method in setup.py.

That said, a library could certainly could update their old releases with a patch release and specify a <= requirement on a particular dependency when versions newer than that no longer work. That said, it would be a bit of work since indirect dependencies would also have to be accounted for as well.

Re: The modern packager’s security nightmare

#189

Post by secabeen » Mon, Feb 22, 2021, 4:54 AM UTC

pdonis wrote: post #175

hctaw wrote: post #42

Earlier quoted context omitted.

> The distribution / maintainer / package manager approach has proven to be an extremely reliable way to get trustworthy software. Many of us love it and want to see it stick around. I disagree, it's proven to be inadequate for modern software development and that's why these new languages/ecosystems are springing up. The least reliable way to package and distribute software is by relying on traditional package manag…

>I disagree, it's proven to be inadequate for modern software development You're not disgreeing with the post you responded to; you're just stating a different priority. hctaw said the distro/maintainer/pm approach is extremely reliable at producing trustworthy software . That means trustworthy to the user . It says nothing at all about how hard producing that software is for the developer. You are saying that the di…

> And anyway, as a user, I don't care. I want my software to work, to be stable, and to not have security flaws; and if a security flaw is found, I want a fix to be pushed to me ASAP. The distro/maintainer/pm approach does that. If instead I have umpteen zillion different statically linked applications installed, each of which packages all of its own dependencies, then instead of just relying on my distro to push security fixes to shared libraries that everyone uses, I have to rely on every single one of those developers to do it for their own packages. And most of them won't do it, or they'll do it when they get around to it instead of when I, the user, need it.

This is my main worry as well. We are currently in the early, easy, period of this new development paradigm, where developers are constantly releasing new code, and fixes are easy to deploy. I worry that in 10 or 15 years, when a security bug is found in a critical imported function, the developers aren't going to be around anymore to fix them, they will have moved on to the next new, hot, language, and will have as much interested in maintaining their old Go/Rust code as developers today do in maintaining their old C code.

Re: The modern packager’s security nightmare

#190

Post by nightowl_games » Mon, Feb 22, 2021, 5:01 AM UTC

bscphil wrote: post #13

Over the last several weeks I was working on an essay about this exact problem, including the connection between static linking and bundling. This one is so well done that I probably won't even publish it. But I'll add this, for people who may not immediately see why this is important. I think that the real danger these new technologies represent is not inherently bad technology, but the possibility of ecosystem dama…

I sympathize with your concern.

Fragmentation is fundamental to Open Source. Fragmentation brings mutation, and mutation brings evolution. Fragmentation is a virtue of open source. Not a vice. The world simply doesn't value security above convenience, as I intuit you may.

I think the natural tendency of all organic evolution is to take the path of least resistance.

It will not be effective to ask developers to do more work. That is going against nature. The packaging ecosystem has to compete and win against its competitors. That is the true way of Open Source, of organic systems.

It is likely that the packaging system you are advocating for is more difficult than the competitors you mentioned. Reducing that friction is perhaps a more effective place to apply your focus. What are these competing ecosystems doing that makes them more attractive? Why is your ecosystem losing users? How does Flathub, Snapcraft and nixOS fit into this view? How does the mac app store, the windows store fit into this view? What is the one true way to package an application?