Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The modern packager’s security nightmare

blogs.gentoo.org

Post reply on HN
Print view View original

131–140 of 282 posts

Re: The modern packager’s security nightmare

#131

Post by tialaramex »

bscphil wrote: post #76
jauer wrote: post #58

Earlier quoted context omitted.

> Two different communities create two very different kinds of software, which run on the same systems, but are created and distributed in very different ways. This is where the disconnect is coming from. The distro maintainers are coming from a world of multi-user systems where backwards compatibility and updating deps without disturbing a user's workload / forcing them to recompile is paramount. Go (and a fair amou…

I agree. I think the difference has been that until recently "the land of CI/CD" and so on has been certain segments of the corporate world, and not how typical open source developers did things*. So when the former developed new technologies and new languages, they created build tools for them that anticipated being used in the ways that they usually produce software. The "problem", in the sense that it's a problem,…

Although continuous integration starts in proprietary software, it's been present in Free Software for at least decades. Netscape may well be the second or third medium-large software outfit to do continuous integration the way it's done today (we know Microsoft had a team doing this by hand every single day for Windows NT but that's completely insane) because some of its team had experienced this approach elsewhere and knew they needed it if they wanted to ship software that actually works. When Mozilla was created, Tinderbox (that system) along with the Mozilla browser (and so today Firefox) and Bugzilla (a bug tracker) were freed.

I know it probably seems like last week, but that was more than twenty years ago.

Re: The modern packager’s security nightmare

#132

Post by Ericson2314 »

bscphil wrote: post #76
jauer wrote: post #58

Earlier quoted context omitted.

> Two different communities create two very different kinds of software, which run on the same systems, but are created and distributed in very different ways. This is where the disconnect is coming from. The distro maintainers are coming from a world of multi-user systems where backwards compatibility and updating deps without disturbing a user's workload / forcing them to recompile is paramount. Go (and a fair amou…

I agree. I think the difference has been that until recently "the land of CI/CD" and so on has been certain segments of the corporate world, and not how typical open source developers did things*. So when the former developed new technologies and new languages, they created build tools for them that anticipated being used in the ways that they usually produce software. The "problem", in the sense that it's a problem,…

I like Nixpkgs and NixOS for understanding both worlds. The Nix ecosystem's best path to mainstream success is being that go-between for everyone.

Re: The modern packager’s security nightmare

#133

Post by tgbugs »

sagichmal wrote: post #117
AshamedCaptain wrote: post #98

Earlier quoted context omitted.

> Dependencies are far more likely to rot because they change in incompatible ways than the underlying hardware does Yes, that is true. It is also very likely that you can more easily go back to a previous version of a dependency than you can go back to a previous hardware. The argument is that, therefore, pinning can only speed up your rotting. If you don't statically link your dependencies, and due to an upgrade so…

Honest question: have you ever worked as an application developer? Responsible for getting working artifacts to users as a means to an end? Pinning dependencies absolutely and unquestionably works better, and for longer, than dynamic linking, for this use case.

If developers are unwilling to maintain dependencies and be good citizens of the larger language community, should they be adding those dependencies in the first place?

If you're not operating in the large ecosystem then fine. But if your project is on e.g. pypi, then there is an issue.

(edit: Note, yes I know the virtualenvs exist, docker exists, etc. but those are space and complexity trade-offs made as a workaround for bad development practices)

Re: The modern packager’s security nightmare

#134

Post by Guvante »

baybal2 wrote: post #115
jauer wrote: post #58

Earlier quoted context omitted.

> Two different communities create two very different kinds of software, which run on the same systems, but are created and distributed in very different ways. This is where the disconnect is coming from. The distro maintainers are coming from a world of multi-user systems where backwards compatibility and updating deps without disturbing a user's workload / forcing them to recompile is paramount. Go (and a fair amou…

> it's easier to reason about code that is running if you can look at the commit that a bin was built from and know exactly what's inside (including all deps). Believe me, it's usually the opposite. Lack of proper releases, testing, and versioning results in unending checkout-fu to figure out what commits for each of 20 libraries will work for each other. The idea is plainly stupid, without any redeeming qualities. T…

The old way of forever support for all versions and never any breaking changes is really beneficial on the consumer side.

However it is incredibly expensive on the producer side.

Package managers have pushed that pain to consumers. It isn't terribly surprising that producers are jumping on board en-masse.

Re: The modern packager’s security nightmare

#135

Post by AshamedCaptain »

sagichmal wrote: post #117
AshamedCaptain wrote: post #98

Earlier quoted context omitted.

> Dependencies are far more likely to rot because they change in incompatible ways than the underlying hardware does Yes, that is true. It is also very likely that you can more easily go back to a previous version of a dependency than you can go back to a previous hardware. The argument is that, therefore, pinning can only speed up your rotting. If you don't statically link your dependencies, and due to an upgrade so…

Honest question: have you ever worked as an application developer? Responsible for getting working artifacts to users as a means to an end? Pinning dependencies absolutely and unquestionably works better, and for longer, than dynamic linking, for this use case.

Perhaps I still fail to explain myself: what I am saying is that _not pinning_ only _adds_ more choices, so by definition it can only work better.

Pinned or not, if a software update breaks things, you can always just revert back to a previous version of your dependencies. This applies to a myriad soft problems including a dependency changing interface.

However, when pinning, when one of your static dependencies is broken due to a change outside your control (e.g. hardware, operating system, security issue making it unusable, or something else), the user's only recourse is to call the developer to fix the software.

I am not claiming whether one happens more frequently than the other, or claiming that hardware changes cannot break the main software itself, which often nullify the point. All these issues can happen to both software with static linking or dynamic linking. However dynamic linking has at least one extra advantage that static linking cannot have, and the opposite is not true.

> have you ever worked as an application developer? Responsible for getting working artifacts to users as a means to an end?

Look, ironically I find that all of this crap discussion is because of a newer generation of "application developers" that do not know yet what does it mean to "deliver working artifacts to users". Imagine my answer to that question.

Re: The modern packager’s security nightmare

#136

Post by Groxx »

syllogism wrote: post #20

This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…

This is the difference between an "application" and a "library": https://caremad.io/posts/2013/07/setup-vs-requirement/

Libraries should absolutely not pin their dependencies. Applications should if you care about reproducible builds (not necessarily byte-for-byte, but "can build today == can build tomorrow").

Installing both libraries and applications in the same way in the same environment is a fundamental mismatch that pip encourages, and yes - it leads to fragile binaries.

Re: The modern packager’s security nightmare

#137

Post by erik_seaberg »

hctaw wrote: post #42
bscphil wrote: post #13

Over the last several weeks I was working on an essay about this exact problem, including the connection between static linking and bundling. This one is so well done that I probably won't even publish it. But I'll add this, for people who may not immediately see why this is important. I think that the real danger these new technologies represent is not inherently bad technology, but the possibility of ecosystem dama…

> The distribution / maintainer / package manager approach has proven to be an extremely reliable way to get trustworthy software. Many of us love it and want to see it stick around. I disagree, it's proven to be inadequate for modern software development and that's why these new languages/ecosystems are springing up. The least reliable way to package and distribute software is by relying on traditional package manag…

Are you saying Debian stable isn’t slow and careful enough in your experience? I usually hear just the opposite.

Re: The modern packager’s security nightmare

#138

Post by cycloptic »

steveklabnik wrote: post #100
archi42 wrote: post #89

Earlier quoted context omitted.

What's the problem with focusing on upstreaming the patches and getting rid of rust's "staging fork"? Especially if they fix something as serious as miscompilations, wouldn't those patches be VERY much needed upstream? I'm asking out of genuine interest.

Quoting from the documentation I linked: > Strongly prefer to upstream all patches to LLVM before including them in rustc. That is, this is already the case. We don't like maintaining a fork. We try to upstream as much as we can. But, at the same time, even when you do this, it takes tons of time. A contributor was talking about exactly this on Twitter earlier today, and estimated that, even if the patch was written…

Can't there be a build option to not use the LLVM submodule, and instead use the system LLVM? Assuming there are tests for these LLVM bugs, and assuming the patches are indeed being merged, wouldn't a CI be able to catch when it is safe for downstream users that want to use upstream LLVM to update their Rust installation?

Re: The modern packager’s security nightmare

#139

Post by skybrian »

sanxiyn wrote: post #65
posix_me_less wrote: post #19

Here is the solution to this Debian/Gentoo/ packager's dilemma. Packagers realize that their numbers are small and they can't keep up fixing all the modern big software (BS) projects that go against their philosophy. They define the core of the OS that they can keep managing in this traditional way (kernel, basic userland, basic desktop, basic server services, basic libraries, security support for all that). "But peo…

> basic desktop I actually wish this is feasible, but it isn't. GNOME is basic desktop, GNOME includes librsvg, librsvg build-depends on Rust. So you need to package Rust (hence LLVM) to have GNOME. Your proposal only works if "core OS" refuses to use Rust.

I think it's likely that there will be "system Rust" which is used to compile libraries used by the OS, but this is independent of the toolchain and crates that most developers use for their own development.

In a binary distribution, you don't necessarily need to install the system toolchain, unless you want to work on the system.

Re: The modern packager’s security nightmare

#140

Post by [deleted] »

GartzenDeHaes wrote: post #36
dataflow wrote: post #33

Why do only *nix folks pull their hair out about this? On Windows programs bundle their dependencies all the time (sometimes even as shared libraries! but without the independent update benefits) and hardly anybody loses sleep over it. Heck, users actually like the fact that it minimizes friction. Nobody claims it's rock-solid security, but does it need to be? Actually, now that I wrote it above, I think I might have…

The unix culture comes from a shared multi-user perspective, whereas the windows point of view tends to be towards a single user desktop. In a shared environment, it's not acceptable for applications to change shared system components. Although this is maybe less important today, the culture still persists.

[deleted]
Post reply on HN