Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I've compared nearly all Rust crates.io crates to contents of their Git repos

mastodon.social

Post reply on HN
Print view View original

11–20 of 65 posts

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#11

Post by KolmogorovComp »

corytheboyd wrote: post #8

How crazy would it be to have a package repository that also builds the artifacts it distributes? You’d need a high barrier to entry to save on costs and time sifting through garbage. Perhaps it’s this high barrier that would prevent such a repository from taking off though. Perhaps this is just a really dumb step on a path leading back to simple checksum validations… though with those, you’re only validating that wh…

crates.io already builds the artefacts.

But the code-source that is sent to crates.io is not necessarily the same as the one in the public repo linked to the crate.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#12

Post by progval »

grahar64 wrote: post #7

Deterministic compilation is the best way to let people validate what they are downloading from repositories is what is in the codebases.

crates.io does not host compiled artifacts. If packages on crates.io differ from their Git repository it's because of a custom pre-build step of that particular package, so a deterministic compilation toolchain won't help here.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#13

Post by progval »

KolmogorovComp wrote: post #11
corytheboyd wrote: post #8

How crazy would it be to have a package repository that also builds the artifacts it distributes? You’d need a high barrier to entry to save on costs and time sifting through garbage. Perhaps it’s this high barrier that would prevent such a repository from taking off though. Perhaps this is just a really dumb step on a path leading back to simple checksum validations… though with those, you’re only validating that wh…

crates.io already builds the artefacts. But the code-source that is sent to crates.io is not necessarily the same as the one in the public repo linked to the crate.

Do you have a source for crates.io building artefacts? I have a couple of crates on it and never saw any sign it tried to compile them, even when they were broken.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#14

Post by [deleted] »

corytheboyd wrote: post #8

How crazy would it be to have a package repository that also builds the artifacts it distributes? You’d need a high barrier to entry to save on costs and time sifting through garbage. Perhaps it’s this high barrier that would prevent such a repository from taking off though. Perhaps this is just a really dumb step on a path leading back to simple checksum validations… though with those, you’re only validating that wh…

[deleted]

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#15

Post by corytheboyd »

KolmogorovComp wrote: post #11
corytheboyd wrote: post #8

How crazy would it be to have a package repository that also builds the artifacts it distributes? You’d need a high barrier to entry to save on costs and time sifting through garbage. Perhaps it’s this high barrier that would prevent such a repository from taking off though. Perhaps this is just a really dumb step on a path leading back to simple checksum validations… though with those, you’re only validating that wh…

crates.io already builds the artefacts. But the code-source that is sent to crates.io is not necessarily the same as the one in the public repo linked to the crate.

Ah yeah, I suppose that’s what I really mean, a means of verifying builds link to source that is publicly available. Sounds like the source repository has to be in on it too

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#16

Post by miki123211 »

xpe wrote: post #10
grahar64 wrote: post #7

Deterministic compilation is the best way to let people validate what they are downloading from repositories is what is in the codebases.

> Deterministic compilation is the best way to let people validate what they are downloading from repositories is what is in the codebases. "the best way"? Please make the argument for why. To do it properly, you must steel-man the alternatives (not shoot down straw-men)

Because deterministic compilation lets you (or someone else) do this automatically.

If you introduce a backdoor into the compilation step, you run a much greater risk of detection. As long as there are multiple machines compiling packages and verifying whether the checksums match, any single backdoored machine will immediately be caught. This is much more important for package managers that do their own builds and ship their own binaries than for those who just ship whatever they got from the developer.

Without deterministic compilation, two builds of the exact same code might differ. This makes backdoors very hard to detect unless you have prior suspicion that one is present in a particular program.

Deterministic compilation forces people to embed backdoors directly in the source code repository, which creates an audit trail, is very visible in diffs, much easier to catch in reviews and so on. You can still get away with it (see the XZ situation), but it requires far more work.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#17

Post by GauntletWizard »

kichimi wrote: post #9
corytheboyd wrote: post #8

How crazy would it be to have a package repository that also builds the artifacts it distributes? You’d need a high barrier to entry to save on costs and time sifting through garbage. Perhaps it’s this high barrier that would prevent such a repository from taking off though. Perhaps this is just a really dumb step on a path leading back to simple checksum validations… though with those, you’re only validating that wh…

Isn't this gentoo?

No, Gentoo does something far from it - it builds everything on the host machine every time, more or less.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#18

Post by thayne »

jonahx wrote: post #4
jraph wrote: post #2

Good initiative. Now people need to go through this and do the reviews :-) Next step would be to do reproducible builds (if it's not already the case).

Not a Rust dev so maybe a dumb question, but is this more involved than just running diffs? If so, what needs to be done?

Most of the diffs are probably innocuous. I suspect the most common diff would be the version line of Cargo.toml, both from CI that automatically updates that line, and people who forgot to update it before making a tag in git.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#19

Post by miki123211 »

corytheboyd wrote: post #8

How crazy would it be to have a package repository that also builds the artifacts it distributes? You’d need a high barrier to entry to save on costs and time sifting through garbage. Perhaps it’s this high barrier that would prevent such a repository from taking off though. Perhaps this is just a really dumb step on a path leading back to simple checksum validations… though with those, you’re only validating that wh…

Distro repositories (like the one you have on Debian / Ubuntu / Redhat etc) do this.

They work on a different model, where only packages that are deemed "worthy" are included, and there's a small-ish set of maintainers that are authorized to make changes and/or accept change requests from the community. In contrast, programming language package managers like cargo, pip or npm let anybody upload new packages with little to no prior verification, and place the responsibility of maintaining them solely on their author.

The distribution way of doing things is sometimes necessary, as different distributions have different policies on what they allow in their repositories, might want to change compilation options or installation paths, backport bug and security fixes from newer project versions for compatibility, or even introduce small code changes to make the program work better (or work at all) on that system.

One example of such a repository, for the Alpine Linux distribution, is at https://github.com/alpinelinux/aports

Post reply on HN