Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The modern packager’s security nightmare

blogs.gentoo.org

Post reply on HN
Print view View original

101–110 of 282 posts

Re: The modern packager’s security nightmare

#101

Post by forrestthewoods »

sigjuice wrote: post #73
hctaw wrote: post #50

The day I stop static linking is the day I can compile on one distro and ship to many without worrying about users reporting loader errors. That day is not today. Until then, I'll keep doing it, because it saves me time and money. I don't really buy that dynamic linking all the things is such a boon to security. But I'll link to this: https://drewdevault.com/dynlib > Not including libc, the only libraries which had "…

The day I stop static linking is the day I can compile on one distro and ship to many without worrying about users reporting loader errors. That day is not today. Maybe do the trendy thing and link your whole distro together with your app (a.k.a containers)?

The fact that containers exist and are prevalent is a damning indictment on the Linux model imho.

Producing software that can simply launch is so wildly complex that the solution is... to snapshot an entire operating system install to run the application.

That’s INSANE. But it’s unfortunately required. https://xkcd.com/1987/

> Now, for the worst of all — one that combines all the aforementioned issues, and adds even more. Bundling (often called vendoring in newspeak) means including the dependencies of your program along with it. The exact consequences of bundling vary depending on the method used.

Let’s consider this statement at face value. The WORST thing a program can do is... include the things the program needs to run? So programs should... NOT include the things they REQUIRE to run?

I get it. I understand the argument. But if you take just a half step back it should be clear how utterly broken the whole situation is.

Re: The modern packager’s security nightmare

#102

Post by zxcvbn4038 »

If your package management is worth it’s salt then you don’t care about static linking, you just update every package that links to that package. IBM’s AIX had the best package management ever IMO, you could roll forward or back and things would just work. Completely deterministic. On the backend whenever a package was updated they would regression test it and everything that depended on it to ensure the update didn’t break anything - when a bug did get through they would fix it and add another regression test so it didn’t happen again. All of that kinda broke when they adopted RPM as a secondary format because the RPMs didn’t make the same guarantees.

One of the best features of Golang is being able to cut a single static linked Linux binary that works on Alpine, Red Hat, CentOS, etc. With Go your also not linking to OpenSSL or Libxml, and those two packages are responsible for every emergency roll out I’ve ever had to do.

Time and again end result of version pinning is that when forced to update developers are required to advance through multiple major version updates and what should be a patch becomes a rewrite. I have to constantly deal with a bunch of web sites on jquery 2 and the developers boo-hooing every day because jquery 3.5 is completely different and all I can tell them is that jquery 2 is no longer maintained so they need to stop using it for new projects and they need to update or retire their existing projects using it.

One of the things I liked about Golang was that it didn’t have any versions to pin so it avoided all of that nonsense altogether. However then they added “modules” and it’s getting harder to not use those, so that killed that magic. Though I did make sure the CI/CD pipelines I use do a go get -u so hopefully that will force the local Go devs to keep their code fresh.

Re: The modern packager’s security nightmare

#103

Post by archi42 »

steveklabnik wrote: post #100
archi42 wrote: post #89

Earlier quoted context omitted.

What's the problem with focusing on upstreaming the patches and getting rid of rust's "staging fork"? Especially if they fix something as serious as miscompilations, wouldn't those patches be VERY much needed upstream? I'm asking out of genuine interest.

Quoting from the documentation I linked: > Strongly prefer to upstream all patches to LLVM before including them in rustc. That is, this is already the case. We don't like maintaining a fork. We try to upstream as much as we can. But, at the same time, even when you do this, it takes tons of time. A contributor was talking about exactly this on Twitter earlier today, and estimated that, even if the patch was written…

Ok, maybe I'm just to spoiled by my previous experience then. I never had any problems getting patches upstreamed relatively quickly, but then e.g. meson or picolibc are much smaller than LLVM and patches are simpler to reason about (I know some compiler construction).

Thanks for the answer.

Re: The modern packager’s security nightmare

#104

Post by rsj_hn »

bscphil wrote: post #47
hctaw wrote: post #42

Earlier quoted context omitted.

> The distribution / maintainer / package manager approach has proven to be an extremely reliable way to get trustworthy software. Many of us love it and want to see it stick around. I disagree, it's proven to be inadequate for modern software development and that's why these new languages/ecosystems are springing up. The least reliable way to package and distribute software is by relying on traditional package manag…

> I disagree, it's proven to be inadequate for modern software development Well, that's exactly why the OP (and my essay) are "anti" modern software development in many ways. The view is that we're moving away from the traditional open source ecosystem and methods of software development with these new technologies, which (to be clear) are good technologies, but were created mostly to solve problems that some large c…

The problem is that the modern practices are being adopted to meet the changing needs of this code, which weren't an issue in the enthusiast linux and BSD communities, but are an issue when open source code is being put in mission critical professionally managed production environments on AWS, or into cars and appliances.

The userland is much more complex now, so freezing dependencies and bundling them in order to have a smaller number of test cases may not have been necessary when the environment was smaller and simpler. The reliability expectations were lower, and there weren't big money professional support contracts which required you to validate your application against a set of well defined environments. All that has changed.

Re: The modern packager’s security nightmare

#105

Post by sanxiyn »

bscphil wrote: post #86
hctaw wrote: post #74

Earlier quoted context omitted.

I'd disagree that the problem of large organizations are different from the problems of the FOSS ecosystem. Organizations just have a financial incentive to fix them, the FOSS ecosystem does not. If mutually incompatible dependencies and security updates breaking software weren't problems for both corporate and FOSS ecoystems, these new technologies wouldn't have needed to exist. They'd just use the existing platform…

> They should be cut out of the loop. I disagree. This is how you get the Google Play store or the "freeware" app marketplace. It sucks. As a user, I'm quite happy to continue using a traditional distribution even if it means I don't get to use a handful of flashy programs by developers that disagree with the concept of maintainers. So far that choice has been much more positive than negative for me, and I'm doing my…

> a handful of flashy programs

This only works if "core" refuses to use Rust, so it's not sustainable. We already can't build Firefox and GNOME without Rust. Maybe Apache and curl next. And then?

Re: The modern packager’s security nightmare

#106

Post by UncleEntity »

IshKebab wrote: post #18

This article doesn't make a good case against static linking, and the author doesn't seem to understand what vendoring is either: > Bundling (often called vendoring in newspeak) means including the dependencies of your program along with it. No, vendoring means including a copy of the source code of dependencies in your repo. You can bundle dependencies without vendoring them. The only argument presented against stat…

> The only argument presented against static linking is that when a library is updated rebuilding dependants takes longer (and people will have to download bigger updates but I doubt many people care about that).

Not really, the argument is that instead of rebuilding one library you now have to rebuild hundreds of applications to see the benefits for whatever the library update was for...assuming upstream even cares enough to bump the version number for the who knows how many libs they are vendoring because they have other stuff to do which is way more important.

Re: The modern packager’s security nightmare

#107

Post by nextos »

kuratkull wrote: post #70
nextos wrote: post #51

Earlier quoted context omitted.

Some of the things you mention are not incompatible with package managers. Have you considered Nix? I actually agree with the parent post in terms of strongly preferring package managers to other means of distributing software. I've always found Linux much easier to admin than other OSes simply because of package managers.

Was really pumped after playing with Nix for a bit. But got bitten twice in the first day. I tried to run my work project using Nix. It needs Pythons python-prctl library - that doesn't work in Nix. So that's a dud. Next I try to use it as a Nim environment - Nim is unable to compile anything in Nix due to some missing (bundled) Glibc symbol. (Nim works nicely in my standard installation). The cool "export full conta…

You can always emulate FHS environments for things that are hard to turn into a Nix package without significant effort.

Aside from that, Guix is an alternative implementation that might have what you need.

Re: The modern packager’s security nightmare

#108

Post by anderskaseorg »

acidbaseextract wrote: post #62
anderskaseorg wrote: post #53

Earlier quoted context omitted.

If you publish a Python library with pinned dependencies, your code is broken as soon as someone tries to use it with another Python library with pinned dependencies, unless you happened to pin exactly the same version of the dependencies you have in common. Python libraries should not pin dependencies. _Applications_ can pin dependencies, including all recursive dependencies of their libraries. There are tools like…

The most frustrating thing is that pip doesn't make it easy to use more loose declared dependencies while freezing to actual concrete dependencies for deployment. Everybody rolls their own. > Python libraries should not pin dependencies. _Applications_ can pin dependencies, including all recursive dependencies of their libraries. Is the pypi package awscli an application or a library? poetry is frustrating in that it…

For this purpose, I’m defining a “library” as any PyPI package that you expect to be able to install alongside other PyPI packages. This includes some counterintuitive ones like mypy, which needs to extract types from packages in the same environment as the code it’s checking.

The awscli documentation recommends installing it into its own virtualenv, in which case pinned dependencies may be reasonable. There are tools like pipx to automate that.

Though in practice, there are reasons that installing applications into their own virtualenv might be inconvenient, inefficient, or impossible. And even when it’s possible, it still comes with the risk of missing security updates unless upstream is doing a really good job of staying on top of them.

I don’t think that respecting declared dependency bounds is a Poetry bug. Pip respects them too (at least as of 20.3, which enables the new resolver by default: https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip...). If a package declares unhelpful bounds, the package should be fixed. (And yes, that means its maintainer might have to deal with some extra issues being filed—that’s part of the job.)

Re: The modern packager’s security nightmare

#109

Post by john-shaffer »

dbaupp wrote: post #46
syllogism wrote: post #20

This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…

Libraries pinning dependencies only fixes a narrow portion of the problem and introduces a bunch of others (particularly in ecosystems where only a single version of a package can exist in a dependency tree). In particular, it is great because it makes life slightly easier for the library developers. However, if every library pinned deps, it becomes much harder to use multiple libraries together: suppose an app used…

Can you give an example of a real ecosystem that can't handle such a conflict? In my actual experience, the package manager will either automatically use the latest version, or in one case has more complex rules but still picks a version on its own (but I stay away from that one due to the surprise factor). Your argument has force against bad package managers and against using very strict dependency requirements, but not against pinning dependencies sensibly in a good ecosystem.

The only conflict I've seen that can't be automatically resolved is when I had some internal dependencies with a common dependency, and one depended on the git repo of the common dep (the "version" being the sha hash of a commit), and another depended on a pinned version of the common dep. Obviously there's no good way to auto-resolve that conflict, so you should generally stick with versions for library deps and not git shas.

Re: The modern packager’s security nightmare

#110

Post by steveklabnik »

archi42 wrote: post #103
steveklabnik wrote: post #100

Earlier quoted context omitted.

Quoting from the documentation I linked: > Strongly prefer to upstream all patches to LLVM before including them in rustc. That is, this is already the case. We don't like maintaining a fork. We try to upstream as much as we can. But, at the same time, even when you do this, it takes tons of time. A contributor was talking about exactly this on Twitter earlier today, and estimated that, even if the patch was written…

Ok, maybe I'm just to spoiled by my previous experience then. I never had any problems getting patches upstreamed relatively quickly, but then e.g. meson or picolibc are much smaller than LLVM and patches are simpler to reason about (I know some compiler construction). Thanks for the answer.

Remember that in my comment above, it’s assuming a speedy review. Release schedules are still a thing; I’m not at my computer anymore, but IIRC, llvm releases once or twice a year, so that’s the inherent limit of making it into a release, not necessarily the review time.
Post reply on HN