Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The modern packager’s security nightmare

blogs.gentoo.org

Post reply on HN
Print view View original

61–70 of 282 posts

Re: The modern packager’s security nightmare

#61

Post by jcranmer »

saurik wrote: post #23
tgbugs wrote: post #7

Fantastic article. I now have something to point people to when they ask "what is wrong with pinning?" or "what is wrong with static linking?" or "why can't you just use pip?" Michał has had to deal with some pretty crazy stuff the last couple of months ... scratch that, years. The recent attempts to empower developers to distribute their own software means that there is now the potential for there to be as many bad…

> I now have something to point people to when they ask "what is wrong with pinning?" or "what is wrong with static linking?" or "why can't you just use pip?" Michał has had to deal with some pretty crazy stuff the last couple of months ... scratch that, years. Literally the only argument in this article against static linking is "it will take an extra couple hours for the distribution to recompile all affected packa…

> maybe the i-cache

This I doubt. On a statically-linked application, all calls to functions are going through a regular function call instruction. When you dynamically link, every call to a function that might cross library boundaries (which on ELF systems defaults to every call that's not to a static function) instead calls into the PLT, which will itself dispatch through to the underlying function call.

Re: The modern packager’s security nightmare

#62

Post by acidbaseextract »

anderskaseorg wrote: post #53
syllogism wrote: post #20

This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…

If you publish a Python library with pinned dependencies, your code is broken as soon as someone tries to use it with another Python library with pinned dependencies, unless you happened to pin exactly the same version of the dependencies you have in common. Python libraries should not pin dependencies. _Applications_ can pin dependencies, including all recursive dependencies of their libraries. There are tools like…

The most frustrating thing is that pip doesn't make it easy to use more loose declared dependencies while freezing to actual concrete dependencies for deployment. Everybody rolls their own.

> Python libraries should not pin dependencies. _Applications_ can pin dependencies, including all recursive dependencies of their libraries.

Is the pypi package awscli an application or a library?

poetry is frustrating in that it doesn't allow you to override a library's declared requirements to break conflicts. They refuse to add support [1][2] for the feature too. awscli for example causes huge package conflict issues that make poetry unusable. It's almost impossible not to run into a requirement conflict with awscli if you're using a broad set of packages, even though awscli will operate happily with a more broad set of requirements than it declares.

[1] https://github.com/python-poetry/poetry/issues/697

[2] https://github.com/python-poetry/poetry/issues/697#issuecomm...

Re: The modern packager’s security nightmare

#63

Post by tgbugs »

posix_me_less wrote: post #41
tgbugs wrote: post #30

Earlier quoted context omitted.

I think that there is a good reason to keep those projects in the main distribution: it makes them more robust and prevents the build systems and bootstrapping process from becoming a massive pile of technical debt. For example, in order to get clojure 10 running on Gentoo I had to go dig through the git history to figure out where spec-alpha and core-specs-alpha and clojure itself did not form a circular dependency…

Do you mean "robust for people using Gentoo" or "robust for everybody"? Isn't the latter responsibility of the Clojure developers rather than packagers?

Robust for everyone. In theory it is the responsibility of the Clojure devs, but if there isn't a packaging workflow that they are aware of, how can they be expected to integrate with it and make design and development decisions that it would reveal to them? The point is more that, the greater variety of different environments a piece of software runs in the more likely it is that systematic weaknesses will be revealed, sort of like chaos engineering, except for whole environments instead of sporadic failures.

Re: The modern packager’s security nightmare

#64

Post by AshamedCaptain »

syllogism wrote: post #20

This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…

> In short unpinned dependencies mean hopeless bit-rot.

No, this is not true, for the simple reason that there will _always_ be unpinned dependencies (e.g. your compiler. your hardware. your processor) and thus _those_ are the ones that will guarantee bitrot.

Pinning a dependency only _guarantees you rot the same or even faster_ because now it's less likely that you can use an updated version of the dependency that supports more recent hardware.

Re: The modern packager’s security nightmare

#65

Post by sanxiyn »

posix_me_less wrote: post #19

Here is the solution to this Debian/Gentoo/ packager's dilemma. Packagers realize that their numbers are small and they can't keep up fixing all the modern big software (BS) projects that go against their philosophy. They define the core of the OS that they can keep managing in this traditional way (kernel, basic userland, basic desktop, basic server services, basic libraries, security support for all that). "But peo…

> basic desktop

I actually wish this is feasible, but it isn't. GNOME is basic desktop, GNOME includes librsvg, librsvg build-depends on Rust. So you need to package Rust (hence LLVM) to have GNOME. Your proposal only works if "core OS" refuses to use Rust.

Re: The modern packager’s security nightmare

#66

Post by paultopia »

I feel like this confuses a lot of things by assuming an extremely sophisticated end user. Sure, if you're a double-threat dev/sysadmin using linux, then when some vulnerability gets discovered in some dynamically linked library on your system, you have the capacity to (a) receive information about that fact, and (b) update it.

But now suppose you're an ordinary person. You use software. Maybe you even have a windows machine.[1] Which is more likely to actually get a security update to you?

(a) You have to update a single piece of software, which you know you've installed, though a recognized distributional channel like an app store or something, and all its dependencies come with it.

(b) You have to either learn what a DLL is and learn how to update it and then hope that nothing you rely on breaks in some mysterious way because of some dependency on a dependency on a dependency on a dependency. Or you have to accept a whole operating system update, assuming that the operating system comes with a DLL update---and hence accepting all of the other crap that comes with operating system updates from Microsoft (and Apple), such as bugginess from complex updates, incompatibilities between new versions of operating systems and software (or hardware) that you rely on, new obnoxious security rules that you might not agree to (looking at you, Cupertino), and massive disruptions to your ability to actually use your computer to do your work.

No thanks.

[1] Maybe this article is specifically targeted against the linux ecosystem? If so, perhaps this issue is ameliorated somewhat, but it still seems to put a fairly substantial burden on end users, that seems to be inconsistent with actually letting non-experts use linux OSes.

Re: The modern packager’s security nightmare

#67

Post by ChrisMarshallNY »

For an OS like Linux, the thing about static linking is 100% spot-on.

However, for vetted systems, like app stores, this makes no difference, as the only dylibs allowed to be patched at runtime, are the OS-supplied (ABI) ones. The whole app needs to be rebuilt, re-vetted, and rereleased.

Frankly, I would be worried about dylibs, but for a different reason, as more than one hack has been done by hijacking dynamic links. I'm trying to remember how it went, but there was once a common practice for Mac programs to deliberately take advantage of this, in order to add functionality.

Re: The modern packager’s security nightmare

#68

Post by beojan »

anotherhue wrote: post #15
choeger wrote: post #9

Earlier quoted context omitted.

The distribution provides support for much longer than upstream does. Also, packaging is actual work. Just because some hipsters in some company consider it cool to release new features every 4 weeks, it does not mean that the volunteers of some linux distribution can keep up with thag. So if you want your distribution to work well, you should focus on workable and transparent interfaces. If you are happy with the su…

> on the other hand you don't need a distribution in the first place. I think you're right about that, I personally want as little distribution as possible. FreeBSD ports or Arch AUR work well for me. The idea of solidifying other people's software into a bundle and then maintaining it (with necessarily limited expertise) seems like a loosing battle.

As someone who uses the AUR a lot, it's really the perfect example of how you just won't update software if it isn't done automatically by the distro package manager, particularly when you use -git packages.

Re: The modern packager’s security nightmare

#69

Post by Ericson2314 »

This is a moronic by saying everything is equally bad.

1. The linking strategy doesn't matter. Just rebuild everything. The reason Rust and Haskell don't do dynamic linking is pervasive inlining. This is required for many abstractions to work well, and asking optimization boundaries to coincide with software origin boundaries is stupid anyways.

The ABI arguement is stupid because replacing all the dynamic libraries with the ABI changes is no worse than rebuilding everything cause static linking. Therefor ignore that part and just think about inlining.

2. Lock files are fine as long as one can in fact bump the minor versions. Yes, we should make tools to enforce minor versions are non-breaking. And it's crazy we didn't have those from day one. But until we do lock files are fine.

3. Vendoring is pure evil.

Re: The modern packager’s security nightmare

#70

Post by kuratkull »

nextos wrote: post #51
hctaw wrote: post #42

Earlier quoted context omitted.

> The distribution / maintainer / package manager approach has proven to be an extremely reliable way to get trustworthy software. Many of us love it and want to see it stick around. I disagree, it's proven to be inadequate for modern software development and that's why these new languages/ecosystems are springing up. The least reliable way to package and distribute software is by relying on traditional package manag…

Some of the things you mention are not incompatible with package managers. Have you considered Nix? I actually agree with the parent post in terms of strongly preferring package managers to other means of distributing software. I've always found Linux much easier to admin than other OSes simply because of package managers.

Was really pumped after playing with Nix for a bit. But got bitten twice in the first day. I tried to run my work project using Nix. It needs Pythons python-prctl library - that doesn't work in Nix. So that's a dud. Next I try to use it as a Nim environment - Nim is unable to compile anything in Nix due to some missing (bundled) Glibc symbol. (Nim works nicely in my standard installation). The cool "export full container" thing mentioned in the tutorial failed for all, even the simplest cases. So I am kinda disillusioned by Nix.
Post reply on HN