Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The modern packager’s security nightmare

blogs.gentoo.org

Post reply on HN
Print view View original

161–170 of 282 posts

Re: The modern packager’s security nightmare

#161

Post by josephg »

bscphil wrote: post #76
jauer wrote: post #58

Earlier quoted context omitted.

> Two different communities create two very different kinds of software, which run on the same systems, but are created and distributed in very different ways. This is where the disconnect is coming from. The distro maintainers are coming from a world of multi-user systems where backwards compatibility and updating deps without disturbing a user's workload / forcing them to recompile is paramount. Go (and a fair amou…

I agree. I think the difference has been that until recently "the land of CI/CD" and so on has been certain segments of the corporate world, and not how typical open source developers did things*. So when the former developed new technologies and new languages, they created build tools for them that anticipated being used in the ways that they usually produce software. The "problem", in the sense that it's a problem,…

> The "problem", in the sense that it's a problem, is that these languages and related technologies are all pretty good!

Yes; and frankly the development ecosystem for making software for linux & friends on top of apt/etc is terrible - at least from the perspective of a modern professional software engineer. The assumption is C, and C programs have no package manager - so of course dependencies get bundled / vendored sometimes when the alternative is linking to potentially out of date dependencies in apt. Autoconf/automake is awful to learn and understand. CMake is better - but its horrendously complicated because it tries to solve the impossible job of paving over all the junky custom compilation scripts that came before. (And it still has no cargo equivalent for actually fetching your deps.)

And then to work around all of that, each distribution will make weird, custom, maybe buggy patches to your software before adding it to their package managers. (Which has caused some high profile bugs and security issues a number of times.) Now when there's a bug, nobody knows who's fault it is!

This worked in a world when there wasn't much software, when releases were rare and when most programs only had one or two dependencies. None of these properties are true any more.

Rust, go, python and nodejs don't fit well with linux's package managers. The obvious alternative would be putting every crate, gem, pip package and npm package into apt, rpm and all the rest. And keeping them up to date with every version. But lets be real - that would be horrible. Apt et al aren't (currently) up to the task. (Can you imagine every npm package needing a maintainer in apt alone? I can just imagine the github issues: "I'm in debian stable and this transitive dep you're using only has version 0.1 available, from 6 years ago. What do I do?". Yikes.)

I'm sympathetic to the argument that modern million dependency software development has its own problems; but right now it (sadly) has no competition in terms of ergonomics and build reliability.

Re: The modern packager’s security nightmare

#162

Post by rcxdude »

hctaw wrote: post #42
bscphil wrote: post #13

Over the last several weeks I was working on an essay about this exact problem, including the connection between static linking and bundling. This one is so well done that I probably won't even publish it. But I'll add this, for people who may not immediately see why this is important. I think that the real danger these new technologies represent is not inherently bad technology, but the possibility of ecosystem dama…

> The distribution / maintainer / package manager approach has proven to be an extremely reliable way to get trustworthy software. Many of us love it and want to see it stick around. I disagree, it's proven to be inadequate for modern software development and that's why these new languages/ecosystems are springing up. The least reliable way to package and distribute software is by relying on traditional package manag…

> I do not trust system provided libraries to function with my applications because I've been burned so many times in the past.

I agree. About half the issues I've had with dependencies has been due to distributions fiddling with upstream for some reason or another. Probably the main reason I like Arch (which has a policy of just following vanilla upstream, though they aren't immune: 'python' being python 3 is probably the biggest pain point)

Re: The modern packager’s security nightmare

#163

Post by cycloptic »

forrestthewoods wrote: post #148
cycloptic wrote: post #129

Earlier quoted context omitted.

You are comparing having containers versus having no containers and no kind of bundling at all, which is not a reasonable comparison. If you need an OS snapshot, the only option on other operating systems is often to use a VM. That's what containers are used as an alternative to. (The other alternative there is to never install patches or upgrades on the production machines, which probably creates as many problems as…

I’m not sure I made that comparison. I’m saying the popularity of containers is an existence proof that the distro-oriented shared library system is failing to meet the needs of common use cases. Programs that bundle dependencies have a radically reduced need for snapshots, VMs, or containers. Such tools do provide a variety of value. But they don’t become virtually requirements to merely launch a program without err…

You can't get rid of the distro-oriented shared library system, it exists as long as you are building on top of an operating system. To that end, when you say "we depend on operating system minimum version X.Y.Z" that now becomes another dependency you have to ship at some point if you're running the production machines. Containers are popular because they actually solve that exact problem. If there was some other comparison you were making I'd be interested to hear it, but AFAIK containers can only help here so it's not clear what else you were comparing to.

Re: The modern packager’s security nightmare

#164

Post by ticviking »

sanxiyn wrote: post #105
bscphil wrote: post #86

Earlier quoted context omitted.

> They should be cut out of the loop. I disagree. This is how you get the Google Play store or the "freeware" app marketplace. It sucks. As a user, I'm quite happy to continue using a traditional distribution even if it means I don't get to use a handful of flashy programs by developers that disagree with the concept of maintainers. So far that choice has been much more positive than negative for me, and I'm doing my…

> a handful of flashy programs This only works if "core" refuses to use Rust, so it's not sustainable. We already can't build Firefox and GNOME without Rust. Maybe Apache and curl next. And then?

We may be forced to work for free to make rusts build less of a pain. Or to abandon those projects which have abandoned us.

Re: The modern packager’s security nightmare

#165

Post by X-Istence »

One of the issues with unbundling everything into its own package really comes to a head when it comes to how Python is packaged on Debian based systems.

They split out core components that are built-in to Python so that they may distribute them separately thereby breaking the Python version's default tooling that is supposed to exist, in the name of "not having dev tools installed alongside non dev".

Package maintainers have made life miserable for those of us who have to then help people through that mess so that they then may use the software we have written and or maintain. We have to write our quick start guides to tell users how to get a proper working version of Python on their system because the package maintainers deem their way of deploying Python the best way to deploy it.

"Your instructions are wrong, `python3 -mvenv venv` doesn't do anything, it just errors out saying I need to install something else, but when I install that package it still doesn't work".

Packagers that insist on splitting every dependency then put the onus back on the community to support it, they don't have to deal with it.

Re: The modern packager’s security nightmare

#166

Post by jholman »

ticviking wrote: post #79
oblio wrote: post #56

Earlier quoted context omitted.

The open source solutions are primarily for C/C++. A bit for Perl, a bit for Python. But they haven't really moved on. Java has been tacked on since forever. Same for .NET, JavaScript, whatever. And if you wanted your program propagated to all major distros you'd have to wait a decade. Nobody has time for that. Not corporations, not mom and pop stores, and I doubt many hobbyists.

Then you always have the option to package your software for the ones you use and not worry about the rest. If you want to make your program popular then packaging is part of the process needed.

> If you want to make your program popular then packaging is part of the process needed.

That is demonstrably not true. I'm not making a value judgment (maybe it SHOULD be true, I dunno), but it's not true. There's a lot of popular unpackaged software out there.

Re: The modern packager’s security nightmare

#167

Post by _y5hn »

forrestthewoods wrote: post #148
cycloptic wrote: post #129

Earlier quoted context omitted.

You are comparing having containers versus having no containers and no kind of bundling at all, which is not a reasonable comparison. If you need an OS snapshot, the only option on other operating systems is often to use a VM. That's what containers are used as an alternative to. (The other alternative there is to never install patches or upgrades on the production machines, which probably creates as many problems as…

I’m not sure I made that comparison. I’m saying the popularity of containers is an existence proof that the distro-oriented shared library system is failing to meet the needs of common use cases. Programs that bundle dependencies have a radically reduced need for snapshots, VMs, or containers. Such tools do provide a variety of value. But they don’t become virtually requirements to merely launch a program without err…

For an entire OS, this doesn't make sense. It does make sense for business critical software. We did this 20 years ago, even calling the subdir VENDOR, for some 3rd party components. It's part of design, and even support of your stuff.

What's new is containers and k8s. People will put more into that, for dev and critical sw.

However, for full desktop or OS it's probably cargo-cult or hard to scale and maintain.

Re: The modern packager’s security nightmare

#168

Post by _y5hn »

noobermin wrote: post #158
mixedCase wrote: post #12

Yeah well, who's hungry? No really, neither users nor developers care, nor should they. I've been using Linux on the desktop for well over a decade and I'm tired of seeing this plea for everything to behave exactly like C or scripting languages because every distro wants to be its own special snowflake and it would be too hard to adapt. The world has changed and distros have to stop pretending it hasn't. Flatpak, Nix…

It's hard not to read this and not feel like a little bit of it is some maintainers worried about losing control. When I get on irc and say the words "pip" they certainly send me those vibes.

It's just idealism, but no ideology suits everyone and for every context. People forget this.

Re: The modern packager’s security nightmare

#170

Post by zrm »

josephg wrote: post #161
bscphil wrote: post #76

Earlier quoted context omitted.

I agree. I think the difference has been that until recently "the land of CI/CD" and so on has been certain segments of the corporate world, and not how typical open source developers did things*. So when the former developed new technologies and new languages, they created build tools for them that anticipated being used in the ways that they usually produce software. The "problem", in the sense that it's a problem,…

> The "problem", in the sense that it's a problem, is that these languages and related technologies are all pretty good! Yes; and frankly the development ecosystem for making software for linux & friends on top of apt/etc is terrible - at least from the perspective of a modern professional software engineer. The assumption is C, and C programs have no package manager - so of course dependencies get bundled / vendored…

> This worked in a world when there wasn't much software, when releases were rare and when most programs only had one or two dependencies. None of these properties are true any more.

I don't think this is even the problem.

It's that upstream maintainers have stopped worrying about compatibility.

Once upon a time you would have regular minor releases of some package. 3.0.2, 3.0.3, 3.0.4, but they were all backwards compatible. If you had version 3.0.4 and some software that was built against 3.0.2, it still worked against 3.0.4 because the only difference was that things were added or compatibly improved, not removed or incompatibly changed.

Version 3.0.x wasn't compatible with version 2.9.x, but then the package maintainer for the distribution only has to package versions 3.0.4 and 2.9.16, i.e. suitably recent minor versions of each compatibility revision. Compatibility revisions so old that nobody relevant uses them anymore can be ignored, so they only had to package two or three incompatible versions which together are compatible with everything in active use.

The problem today is that everything is a compatibility-breaking change, so there are dozens of releases from this year alone that are all mutually incompatible and would have to be packaged separately. And that doesn't scale.

Post reply on HN