Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The modern packager’s security nightmare

blogs.gentoo.org

Post reply on HN
Print view View original

141–150 of 282 posts

Re: The modern packager’s security nightmare

#141

Post by steveklabnik »

cycloptic wrote: post #138
steveklabnik wrote: post #100

Earlier quoted context omitted.

Quoting from the documentation I linked: > Strongly prefer to upstream all patches to LLVM before including them in rustc. That is, this is already the case. We don't like maintaining a fork. We try to upstream as much as we can. But, at the same time, even when you do this, it takes tons of time. A contributor was talking about exactly this on Twitter earlier today, and estimated that, even if the patch was written…

Can't there be a build option to not use the LLVM submodule, and instead use the system LLVM? Assuming there are tests for these LLVM bugs, and assuming the patches are indeed being merged, wouldn't a CI be able to catch when it is safe for downstream users that want to use upstream LLVM to update their Rust installation?

There is. We even test that it builds as part of our CI, to make sure it works, IIRC.

Re: The modern packager’s security nightmare

#142

Post by vnlp »

MetaDark wrote: post #130
mwcampbell wrote: post #113

Earlier quoted context omitted.

This makes me wonder if there's a Linux base system suitable for servers that embraces the newer approach. That is, a minimal base system that's built for immutable container images, providing just what's needed to bootstrap the current generation of language-specific build and package systems. The Alpine Linux Docker images might be a good choice for now, but IIUC, Alpine Linux itself still embraces the older distro…

Maybe the Nix package manager / NixOS is what you're looking for? I think it takes the best features from both worlds. Every package installed with Nix is isolated into content-addressable* directories, so for example, my install of Firefox is located at /nix/store/c7pmng2x05dkigpbhnjs8fdzd8kk31np-firefox-85.0.2/bin/firefox. This is pretty inconvenient to use directly, so Nix generates a profile that symlinks all you…

Sounds like another idea of djb took off (the slashpackage hierarchy).

Re: The modern packager’s security nightmare

#143

Post by cycloptic »

steveklabnik wrote: post #141
cycloptic wrote: post #138

Earlier quoted context omitted.

Can't there be a build option to not use the LLVM submodule, and instead use the system LLVM? Assuming there are tests for these LLVM bugs, and assuming the patches are indeed being merged, wouldn't a CI be able to catch when it is safe for downstream users that want to use upstream LLVM to update their Rust installation?

There is. We even test that it builds as part of our CI, to make sure it works, IIRC.

Hmm, I couldn't find that last time I looked at building Rust -- all I saw was an option to grab precompiled builds of Rust LLVM from the CI server, but maybe I missed something.

Re: The modern packager’s security nightmare

#144

Post by Lx1oG-AWb6h_ZG0 »

jcranmer wrote: post #61
saurik wrote: post #23

Earlier quoted context omitted.

> I now have something to point people to when they ask "what is wrong with pinning?" or "what is wrong with static linking?" or "why can't you just use pip?" Michał has had to deal with some pretty crazy stuff the last couple of months ... scratch that, years. Literally the only argument in this article against static linking is "it will take an extra couple hours for the distribution to recompile all affected packa…

> maybe the i-cache This I doubt. On a statically-linked application, all calls to functions are going through a regular function call instruction. When you dynamically link, every call to a function that might cross library boundaries (which on ELF systems defaults to every call that's not to a static function) instead calls into the PLT, which will itself dispatch through to the underlying function call.

We had an article on the front page yesterday about Swift’s ABI and how much Apple has invested in dynamic linking: https://news.ycombinator.com/item?id=26205969

> It's worth noting that the Swift devs disagree with the Rust and C++ codegen orthodoxy in one major way: they care much more about code sizes (as in the amount of executable code produced). More specifically, they care a lot more about making efficient usage of the cpu's instruction cache, because they believe it's better for system-wide power usage. Apple championing this concern makes a lot of sense, given their suite of battery-powered devices.

Re: The modern packager’s security nightmare

#145

Post by MetaDark »

vnlp wrote: post #142
MetaDark wrote: post #130

Earlier quoted context omitted.

Maybe the Nix package manager / NixOS is what you're looking for? I think it takes the best features from both worlds. Every package installed with Nix is isolated into content-addressable* directories, so for example, my install of Firefox is located at /nix/store/c7pmng2x05dkigpbhnjs8fdzd8kk31np-firefox-85.0.2/bin/firefox. This is pretty inconvenient to use directly, so Nix generates a profile that symlinks all you…

Sounds like another idea of djb took off (the slashpackage hierarchy).

I'm not sure what the slashpackage hierarchy is, but Nix & NixOS started in 2003.

Re: The modern packager’s security nightmare

#146

Post by steveklabnik »

cycloptic wrote: post #143
steveklabnik wrote: post #141

Earlier quoted context omitted.

There is. We even test that it builds as part of our CI, to make sure it works, IIRC.

Hmm, I couldn't find that last time I looked at building Rust -- all I saw was an option to grab precompiled builds of Rust LLVM from the CI server, but maybe I missed something.

https://rustc-dev-guide.rust-lang.org/getting-started.html#c...

links to https://rustc-dev-guide.rust-lang.org/building/suggested.htm...

... which is broken. I'll file a docs bug. Can't do it right this moment, but will in a day or two.

It is not my area of expertise, but I believe the way you do it is to set this option: https://github.com/rust-lang/rust/blob/master/config.toml.ex...

buuut the dockerfile for the llvm 9 build passes this flag as an argument: https://github.com/rust-lang/rust/blob/5a549d36ee81b226d1672...

Re: The modern packager’s security nightmare

#147

Post by lifthrasiir »

MetaDark wrote: post #145
vnlp wrote: post #142

Earlier quoted context omitted.

Sounds like another idea of djb took off (the slashpackage hierarchy).

I'm not sure what the slashpackage hierarchy is, but Nix & NixOS started in 2003.

That's https://cr.yp.to/slashpackage.html which is dated at least 2001. It's not quite the same to the Nix approach though, which also tracks all direct and transitive dependencies.

Re: The modern packager’s security nightmare

#148

Post by forrestthewoods »

cycloptic wrote: post #129
forrestthewoods wrote: post #101

Earlier quoted context omitted.

The fact that containers exist and are prevalent is a damning indictment on the Linux model imho. Producing software that can simply launch is so wildly complex that the solution is... to snapshot an entire operating system install to run the application. That’s INSANE. But it’s unfortunately required. https://xkcd.com/1987/ > Now, for the worst of all — one that combines all the aforementioned issues, and adds even…

You are comparing having containers versus having no containers and no kind of bundling at all, which is not a reasonable comparison. If you need an OS snapshot, the only option on other operating systems is often to use a VM. That's what containers are used as an alternative to. (The other alternative there is to never install patches or upgrades on the production machines, which probably creates as many problems as…

I’m not sure I made that comparison. I’m saying the popularity of containers is an existence proof that the distro-oriented shared library system is failing to meet the needs of common use cases.

Programs that bundle dependencies have a radically reduced need for snapshots, VMs, or containers. Such tools do provide a variety of value. But they don’t become virtually requirements to merely launch a program without error.

Ship your damn dependencies says I. Either statically or dynamically, but ship them.

Re: The modern packager’s security nightmare

#149

Post by mroche »

mwcampbell wrote: post #113
jauer wrote: post #58

Earlier quoted context omitted.

> Two different communities create two very different kinds of software, which run on the same systems, but are created and distributed in very different ways. This is where the disconnect is coming from. The distro maintainers are coming from a world of multi-user systems where backwards compatibility and updating deps without disturbing a user's workload / forcing them to recompile is paramount. Go (and a fair amou…

This makes me wonder if there's a Linux base system suitable for servers that embraces the newer approach. That is, a minimal base system that's built for immutable container images, providing just what's needed to bootstrap the current generation of language-specific build and package systems. The Alpine Linux Docker images might be a good choice for now, but IIUC, Alpine Linux itself still embraces the older distro…

> That is, a minimal base system that's built for immutable container images, providing just what's needed to bootstrap the current generation of language-specific build and package systems.

At least for the first half, that sounds sort of like Fedora/Red Hat CoreOS[0], the predecessor CoreOS fork Flatcar Container Linux[1], or the Amazon distribution BottleRocket[2].

[0] https://getfedora.org/en/coreos?stream=stable

[1] https://kinvolk.io/flatcar-container-linux/

[2] https://aws.amazon.com/blogs/aws/bottlerocket-open-source-os...

Post reply on HN