Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The modern packager’s security nightmare

blogs.gentoo.org

Post reply on HN
Print view View original

241–250 of 282 posts

Re: The modern packager’s security nightmare

#241

Post by virgo_eye »

progval wrote: post #218
syllogism wrote: post #20

This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…

> If you publish a Python library without pinned dependencies, your code is broken. > you will be unable to today publish an end-to-end set of commands that will work in 2025 Not necessarily. Since ~2010 I maintain an application with an unpinned requirements.txt; it doesn't even have version constraints at all. The only breakages I had were either: 1. when switching from Python 2 to Python 3 (obviously) 2. when a ne…

It's clear that your approach is a possible approach:

1. 'only use good libraries' 2. 'it's not your job as a developer to make sure your application can be installed' 3. 'if your users want to use pip... they should be able to fix those issues'

However, this isn't a solution to the problem that led to the existence of language ecosystems. It is a refusal to acknowledge the problem.

Re: The modern packager’s security nightmare

#242

Post by aragilar »

josephg wrote: post #161
bscphil wrote: post #76

Earlier quoted context omitted.

I agree. I think the difference has been that until recently "the land of CI/CD" and so on has been certain segments of the corporate world, and not how typical open source developers did things*. So when the former developed new technologies and new languages, they created build tools for them that anticipated being used in the ways that they usually produce software. The "problem", in the sense that it's a problem,…

> The "problem", in the sense that it's a problem, is that these languages and related technologies are all pretty good! Yes; and frankly the development ecosystem for making software for linux & friends on top of apt/etc is terrible - at least from the perspective of a modern professional software engineer. The assumption is C, and C programs have no package manager - so of course dependencies get bundled / vendored…

I disagree that rust, go or python inherently do not work with linux package managers, instead there's a culture within subsections of those communities who do not value stability (I'm not sure about nodejs community though), which cause these disagreements. C isn't immune to this (and never has been)—science codes are infamous around their lack of care around stability (and IMHO one of the reasons for scientific python's success over alternatives has been this stability).

I'd also disagree with the idea that the alternative is reliable—try building and modifing a project that hasn't been touched in 6 months, and see how reliable that is. With your tree of dependencies (absent specific projects which abstract over a set of unstable dependencies and hence implicitly stabilise them, e.g. SDL) the stability of your project (whether it is a library or application or framework) is set by the least stable dependency you have. Increasing you dependencies increases the risk, but if you have an ecosystem which values stability, larger dependency trees should not see a significant increase in risk (personally, I'd love to see the scientific rust ecosystem achieve similar stability to python).

That's not to say linux distros are perfect (change can be slow ;)), but there's lots of little things they do get right (how many projects handle updating configuration files correctly—that kind of thing is built into distro tooling), and they enable ecosystem-wide changes more that the alternative (e.g. https://reproducible-builds.org/).

Re: The modern packager’s security nightmare

#243

Post by orra »

ece wrote: post #219
orra wrote: post #77

Earlier quoted context omitted.

> I don't really buy that dynamic linking all the things is such a boon to security. Agreed. It's really not a panacea. When you upgrade a library, you probably want to restart the running applications that depend upon this. Dynamic linking won't save you. The solution is having a graph of your dependencies! IIRC, NixOS gets this right. I don't think Debian's apt did?

There are scripts and daemons that help you determine what needs restarting[1,2]. NixOS installs can go in separate directory prefixes when there are conflicts. For Gentoo and other Linux distributions, maintainers usually won't mark something stable without resolving conflicts, and this usually means sticking to older stable version of libraries until newer versions of libraries are fully supported by all installed…

Thanks, checkrestart looks useful. I wasn't aware of it.

Given that it's in a goodies package though, I assume it's not integrated with apt—at least by default.

I'm thinking of that apt prompt that says "There are services installed on your system which need to be restarted when certain libraries, such as libpam, libc, and libssl, are upgraded." I presume then that uses a hardcoded list of important services.

Re: The modern packager’s security nightmare

#244

Post by drran »

lifthrasiir wrote: post #232
drran wrote: post #212

Earlier quoted context omitted.

Distributions are not breaking software, they are just distributing broken software. Blame upstream for lack of testing. Or create your own distribution of flawless software and keep it up to date and flawless for the rest of your life.

Oh, sure. Softwares are broken, you can't fix that. Traditional distros just happen to (falsely) believe that they can somehow fix at their level. What we need instead is a distro that is resilient to software breakage, not flawless software. Does my hope look that unreasonable to you?

My distro works just fine, when an X11 app crashes due to a bug in the Nvidia driver, but I cannot.

Re: The modern packager’s security nightmare

#245

Post by thu2111 »

jjav wrote: post #192
hctaw wrote: post #42

Earlier quoted context omitted.

> The distribution / maintainer / package manager approach has proven to be an extremely reliable way to get trustworthy software. Many of us love it and want to see it stick around. I disagree, it's proven to be inadequate for modern software development and that's why these new languages/ecosystems are springing up. The least reliable way to package and distribute software is by relying on traditional package manag…

> The least reliable way to package and distribute software is by relying on traditional package managers. Checks and balances produce reliability. Having a packaging process and people distinct from the code author validating and enforcing it, most certainly produces much more reliable, secure and stable packages and distributions. Sure, it's slower. Slower is actually a feature in this use case. Having anyone push…

It really doesn't. I've worked on upstream projects where a significant fraction of all bugs reported were created by distributions screwing up packaging and patching in ways they weren't at all qualified to understand and frequently led to non-obvious failures.

When we tried to work with them to fix this, about half the time they flamed us and quoted distro 'policy' as a reason not to fix their bugs, so we just refused to accept bug reports from anyone using those packages anymore.

The fact is that a lot of old-school Linux distributions are built by people who have only a very vague understanding of the software they're packaging, and frequently are closer to the sysadmin side of things than the large-scale software development side. It makes the relationships very frustrating and that's why proprietary software vendors invariably opt-out of distro packaging. Even with apps statically linked to the max possible level Linux users generate disproportionate levels of support tickets due to the general flakiness of the distros they use, so allowing them to modify tested software even further is a losing proposition.

Basically the whole concept of a Linux distribution is obsolete, fading away and irretrievably broken. Hence the proliferation of containers.

Re: The modern packager’s security nightmare

#246

Post by kasabali »

orra wrote: post #77
hctaw wrote: post #50

The day I stop static linking is the day I can compile on one distro and ship to many without worrying about users reporting loader errors. That day is not today. Until then, I'll keep doing it, because it saves me time and money. I don't really buy that dynamic linking all the things is such a boon to security. But I'll link to this: https://drewdevault.com/dynlib > Not including libc, the only libraries which had "…

> I don't really buy that dynamic linking all the things is such a boon to security. Agreed. It's really not a panacea. When you upgrade a library, you probably want to restart the running applications that depend upon this. Dynamic linking won't save you. The solution is having a graph of your dependencies! IIRC, NixOS gets this right. I don't think Debian's apt did?

> When you upgrade a library, you probably want to restart the running applications that depend upon this. Dynamic linking won't save you.

That's an orthogonal issue. You need to restart affected services nevertheless, whether it was linked statically or bundled in nix.

I wish people stopped repeating this red herring restart argument against dynamic linking every time the topic is brought up.

Re: The modern packager’s security nightmare

#247

Post by rictic »

zrm wrote: post #176
rictic wrote: post #126

Earlier quoted context omitted.

Are you actually talking about static linking here, or just venting your spleen about sloppiness in software engineering practice more generally? Because tracking dependencies in source control (specifically, checking in lock files) is tremendous for reproducibility. It means that when bisecting through the commit history to find when a problem began is not just bisecting through the local source code, but also the s…

> you're able to find the first commit that introduced the issue in O(log(commits)) time, rather than needing O(commits * (num dependencies * dependency versions)) time. Try the oldest and newest compatible version of your code and the oldest and newest compatible version of each dependency. This is O(num dependencies), which is generally small N and you can often guess which to try first. Now do each version of the…

While version 3 of the dependency is innocent, commit 5678 is not. Something went wrong in the interaction between the code and its dependencies in that change and discovering that change quickly is valuable.

From there you can start stepping through the code, or simplifying the situation to get a minimal repro of what's going on, or even bisecting with dependency version 3 held constant to see if that's diagnostic.

In my experience, with a bug that's so timing sensitive that some cache lines moving around will trigger it, you're likely to discover pretty quickly that something weird is happening as you try to get a more minimal repro.

Meanwhile, if you're not tracking dependency versions, such a test is likely to appear weirdly flaky. You're trying to track down this failure but something else comes up and you have to set it aside for a few days. When you come back to it, you can no longer get the test to fail because an untracked dependency change has moved cache lines around again. Which change? Which dependency? Since it's not in source control, you've got a painful process of guessing plausible combinations of recent versions of all transitive dependencies.

Re: The modern packager’s security nightmare

#248

Post by drran »

josephg wrote: post #236
drran wrote: post #207

Earlier quoted context omitted.

So, you deliberately chose distro intended for users, with low version churn, instead of distro intended for developers, e.g. Fedora, which ships even pre-release versions sometimes, and now you blame ... apt? Just curious, what you are using for coding? MS Word or Excel? For example, Linus uses Fedora/MATE/Emacs. It's relatively easy to convert between packages between different packagers/distros. Automatic converte…

If you want your software to work everywhere, you either need to take responsibility for making it build everywhere (yes, including old versions of Debian which don’t support the versions of your dependencies you need) or you punt that work to someone else - in which case your software simply won’t work on lots of computers. From the perspective of an upstream maintainer, the status quo is pretty awful. There’s a rea…

If you want for your software to work everywhere, but you want to avoid compilation step, then use Perl/Python/PHP with bindings to QT/GTK.

Linux has perfect backward compatibility, I still able to compile and run 70 years old app, developed on completely different OS and processor.

IMHO, you think that your app/lib binary will work on all combinations of OS/processor flawlessly without recompilation, which is not true. Nobody promising that. It's by design.

Re: The modern packager’s security nightmare

#249

Post by ticviking »

lmm wrote: post #197
ticviking wrote: post #85

Earlier quoted context omitted.

> maintainers shouldn't place themselves in the development process. Then perhaps we as developers should spend more time thinking about and participating in the maintenance process?

Which is exactly what developers have done, and what this guy is complaining about: developers have figured out a different approach to maintenance that works better.

I don't pretend to have an answer but I'm trying to listen to both sides so maybe I can formulate "the question" a little bit better and improve the discussion around this.

It seems like have posts every few months where security professionals, maintainers and sysadmins explain that the "developer usability first" approach to maintenance and dependency management is having massive consequences to our ability to keep systems secure, and to even know if an affected version of a library is on a system.

That is a different problem that doesn't seem to be addressed at all in the current iteration of these tools. If you're aware of efforts to solve that problem in the rust and go ecosystems(since those are the ones cited here) I'd love to read about it.

Re: The modern packager’s security nightmare

#250

Post by ticviking »

virgo_eye wrote: post #240
ticviking wrote: post #79

Earlier quoted context omitted.

Then you always have the option to package your software for the ones you use and not worry about the rest. If you want to make your program popular then packaging is part of the process needed.

I mean the whole argument here is that distro packaging is not needed, that there's a better way of doing it.

That actually clarifies some thing about the debate for me. Thanks.

I'm not sure I'm convinced based on the semi-frequent posts from maintainers and security pros about the issues with vendoring dependencies for software that is widely deployed.

This "better way", since we lack a more concrete name. Seems to be really great if you're running a web app, or server software in your own company and can rebuild and run a rolling deploy pretty easily. For someone pushing software to users all over the world, and as one of those users the downside to allow every application to be responsible for updating this stuff seems pretty steep.

Post reply on HN