Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The modern packager’s security nightmare

blogs.gentoo.org

Post reply on HN
Print view View original

201–210 of 282 posts

Re: The modern packager’s security nightmare

#201

Post by lmm »

syllogism wrote: post #20

This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…

> It's maddening to hear people say things like, "Oh if everyone just used semantic versioning this wouldn't be a problem". Of course this cannot work. _Think about it_. There are innumerable ways two pieces of code can be incompatible. You might have a change that alters the time-complexity for niche inputs, making some call time-out that used to succeed. You might introduce a new default keyword argument that throws off a *kwargs. If you call these things "breaking" changes, you will constantly be increasing the major version. But if you increase the major version every release, what's the point of semver! You're not actually conveying any information about whether the changes are "breaking".

That's the point of the link to Hyrum's law. The article argues that the practice of pinning encourages that attitude: consumers feel free to depend on internal implementation details, producers feel free to change behaviour arbitrarily, and no-one takes responsibility for specifying and maintaining a stable interface, which is how you actually break that knot - producers need to specify which parts are stable interfaces and which are not, consumers need to respect that and not depend on implementation details, and then you can actually use semver because it's clear what's a breaking change and what isn't.

Re: The modern packager’s security nightmare

#202

Post by lmm »

delroth wrote: post #59
posix_me_less wrote: post #19

Here is the solution to this Debian/Gentoo/ packager's dilemma. Packagers realize that their numbers are small and they can't keep up fixing all the modern big software (BS) projects that go against their philosophy. They define the core of the OS that they can keep managing in this traditional way (kernel, basic userland, basic desktop, basic server services, basic libraries, security support for all that). "But peo…

You're making a distinction between "packagers" and "users" which does not exist. Packagers are advanced users that take the initiative to improve their distro when they find software they want to use and that isn't integrated in their distro. > Let the new complicated juggernaut software be deployed and configured by the users, according to developers' instructions, using whatever modern mechanism they prefer I'm no…

> The existence of packagers is proof that some subset of users "prefer" that software "be deployed and configured" via distro package managers.

Not necessarily. It might be inertia - back when linux distributions started being organised this way, those language dependency management mechanisms pretty much didn't exist.

Re: The modern packager’s security nightmare

#203

Post by alfiedotwtf »

Foxboron wrote: post #112
alfiedotwtf wrote: post #81

Earlier quoted context omitted.

I’ve been on Debian since Potato, so I totally see what you’re saying. But... > that's cutting maintainers out of the loop Is this necessarily a bad thing? The market has seen the need to fill a hole, and it seems to be working. I first started with Slackware, and dependency nightmares is what got me into Debian in the first place. Although Debian is nice because of its slow and stable base (which makes me happy for…

>With systems like Arch that get us closer and closer to upstream, are maintainers the unnecessary middlemen? Of course they’re not entirely redundant, but maybe a new model of distros like Arch will be more commonplace in the future Arch is an old distribution, very much in the same class as Fedora, Debian, Gentoo and all the traditional ones. What makes you think Arch makes maintainers even slightly redundant? We s…

> We still deal with security issues

I don’t see your point. In fact, you’ll most likely find that distros that follow upstream closer than “slow and stable releases” will get their patches as soon as upstream fixes them

Re: The modern packager’s security nightmare

#204

Post by globular-toast »

orf wrote: post #178
syllogism wrote: post #20

This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…

You’re completely wrong and this advice is somewhat harmful. What you’re describing is how a Python application should be managed. Not a library. Libraries should absolutely not lock their advertised dependencies to arbitrary point-in-time versions for fairly obvious reasons. Picking a suitable dependency specifier depends heavily on the maturity of the library you’re using and if you need any specific features added…

Even applications shouldn't really be pinning dependencies. The only time to pin dependencies is when deploying that application. That could mean bundling it with pyinstaller or making a docker image. But someone should still be able to install it from source with their own dependencies.

Re: The modern packager’s security nightmare

#205

Post by globular-toast »

syllogism wrote: post #20

This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…

Completely and utterly wrong. I hope nobody heeds this advice.

Re: The modern packager’s security nightmare

#206

Post by globular-toast »

syllogism wrote: post #181
orf wrote: post #178

Earlier quoted context omitted.

You’re completely wrong and this advice is somewhat harmful. What you’re describing is how a Python application should be managed. Not a library. Libraries should absolutely not lock their advertised dependencies to arbitrary point-in-time versions for fairly obvious reasons. Picking a suitable dependency specifier depends heavily on the maturity of the library you’re using and if you need any specific features added…

Everyone needs commands like "pip install 'spacy==2.3.5'" to work reliably in the future, so that you can go back and bisect errors. You need to be able to get back to a particular known-good state, and work through changes I'm not saying we pin our dependencies to exact specific versions, but we absolutely do set an upper bound, usually to the minor version.

> I'm not saying we pin our dependencies to exact specific versions, but we absolutely do set an upper bound, usually to the minor version.

OK. That's more sensible, but "pinning" implies == to a specific version. If you know a library does semantic versioning and breaks their API then ~= is fine. Just not ==.

Re: The modern packager’s security nightmare

#207

Post by drran »

josephg wrote: post #161
bscphil wrote: post #76

Earlier quoted context omitted.

I agree. I think the difference has been that until recently "the land of CI/CD" and so on has been certain segments of the corporate world, and not how typical open source developers did things*. So when the former developed new technologies and new languages, they created build tools for them that anticipated being used in the ways that they usually produce software. The "problem", in the sense that it's a problem,…

> The "problem", in the sense that it's a problem, is that these languages and related technologies are all pretty good! Yes; and frankly the development ecosystem for making software for linux & friends on top of apt/etc is terrible - at least from the perspective of a modern professional software engineer. The assumption is C, and C programs have no package manager - so of course dependencies get bundled / vendored…

So, you deliberately chose distro intended for users, with low version churn, instead of distro intended for developers, e.g. Fedora, which ships even pre-release versions sometimes, and now you blame ... apt? Just curious, what you are using for coding? MS Word or Excel? For example, Linus uses Fedora/MATE/Emacs.

It's relatively easy to convert between packages between different packagers/distros. Automatic converters are exists for deb/rpm/pip/cpan/ctan/cargo, so it's easy to convert all existing packages into one packaging system and drop all of them into huge mono-repo.

Yes, it's much easier to throw a new version to repo in many non-linux repositories. It's the equivalent of rolling distros, such as Arch, or development version of distro, such as Rawhide in Fedora or Sid in Debian. However, it also much easier to: break the world and make into news, distribute keyloggers and steal passwords and keys, forget to backport security fixes to users of older versions, pivot into completely orthogonal thing, etc. No code reviews means no responsibility.

Re: The modern packager’s security nightmare

#208

Post by drran »

hctaw wrote: post #42
bscphil wrote: post #13

Over the last several weeks I was working on an essay about this exact problem, including the connection between static linking and bundling. This one is so well done that I probably won't even publish it. But I'll add this, for people who may not immediately see why this is important. I think that the real danger these new technologies represent is not inherently bad technology, but the possibility of ecosystem dama…

> The distribution / maintainer / package manager approach has proven to be an extremely reliable way to get trustworthy software. Many of us love it and want to see it stick around. I disagree, it's proven to be inadequate for modern software development and that's why these new languages/ecosystems are springing up. The least reliable way to package and distribute software is by relying on traditional package manag…

> By statically linking everything as much as possible and shipping everything else in a self contained bundle with a launcher that overrides any symbols that might inadvertently be pulled in from the system.

So, you propose to ship your own OS, as single image, for your application? What stops you from doing this? Drivers? Ship your own hardware then, like smartphone vendors do.

Re: The modern packager’s security nightmare

#209

Post by Foxboron »

alfiedotwtf wrote: post #203
Foxboron wrote: post #112

Earlier quoted context omitted.

>With systems like Arch that get us closer and closer to upstream, are maintainers the unnecessary middlemen? Of course they’re not entirely redundant, but maybe a new model of distros like Arch will be more commonplace in the future Arch is an old distribution, very much in the same class as Fedora, Debian, Gentoo and all the traditional ones. What makes you think Arch makes maintainers even slightly redundant? We s…

> We still deal with security issues I don’t see your point. In fact, you’ll most likely find that distros that follow upstream closer than “slow and stable releases” will get their patches as soon as upstream fixes them

This assumes an ideal upstream: This is not always the case.

If someone publishes a CVE for a Go or Rust library it's not always the case the project is well maintained or the dev cares to update the dependency. Even if they did, there are no guarantees the upstream decides to publish a minor release just to update dependencies. Because that is what vendoring dependencies get you.

Instead of applying one patch to a shared library I'd need to hunt down all upstreams utilizing the library and manually patch between 10-140 packages independently and submit them upstream.

Re: The modern packager’s security nightmare

#210

Post by lmm »

nwallin wrote: post #154
anotherhue wrote: post #4

Correct me if I'm wrong (really) but isn't static linking mostly a problem for packagers? The less a package maintainer modifies the application the better IMO. If application A and B rely on dependency D, which turns out to have a vulnerability, fixed in D', Then _why_ do we think it is anyone but A & B's developers responsibility to update to D' and distribute patched versions? If the packager tries to do it, the c…

> Correct me if I'm wrong (really) but isn't static linking mostly a problem for packagers? The less a package maintainer modifies the application the better IMO. IMHO that's precisely why dependencies should be unpinned. Let's say application A relies on dependencies B, C, and D, and dependencies B, C, and D depend on dependency E. Let's say dependency E has a critical security vulnerability and needs to be updated…

Making sure that there are multiple people with the knowledge and access to produce new releases of E that incorporate security fixes in a timely way is definitely good. But I'm not convinced that distro maintainers are a good answer to that problem; the distro landscape is very fragmented, and distro maintainers are often not very closely involved with the packages they're notionally maintaining or aware of best practices / pitfalls that apply to that ecosystem. I suspect that something along the lines of the rust platform efforts might have a better chance of pushing out releases of all reverse-dependencies of some package that had a security flaw in a timely way, with minimal risk of breakages.
Post reply on HN