Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I've compared nearly all Rust crates.io crates to contents of their Git repos

mastodon.social

Post reply on HN
Print view View original

61–65 of 65 posts

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#61

Post by pabs3 »

progval wrote: post #12
grahar64 wrote: post #7

Deterministic compilation is the best way to let people validate what they are downloading from repositories is what is in the codebases.

crates.io does not host compiled artifacts. If packages on crates.io differ from their Git repository it's because of a custom pre-build step of that particular package, so a deterministic compilation toolchain won't help here.

> crates.io does not host compiled artifacts.

It definitely does contain generated files, at least one crate has Rust code generated by a Python script that is not in the crate, only in the upstream Git repository.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#62

Post by pabs3 »

kibwen wrote: post #26
KolmogorovComp wrote: post #11

Earlier quoted context omitted.

crates.io already builds the artefacts. But the code-source that is sent to crates.io is not necessarily the same as the one in the public repo linked to the crate.

It's possible that crates.io might attempt to build a crate when published as a sort of sanity check (I don't know if this is true, but it's certainly feasible), but it doesn't distribute binaries, it distributes source code.

> it doesn't distribute binaries, it distributes source code.

It definitely does contain generated files, at least one crate has Rust code generated by a Python script that is not in the crate, only in the upstream Git repository.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#63

Post by xpe »

xpe wrote: post #57
miki123211 wrote: post #16

Earlier quoted context omitted.

Because deterministic compilation lets you (or someone else) do this automatically. If you introduce a backdoor into the compilation step, you run a much greater risk of detection. As long as there are multiple machines compiling packages and verifying whether the checksums match, any single backdoored machine will immediately be caught. This is much more important for package managers that do their own builds and sh…

Ok, but you didn't talk about alternatives. Why not just checksum or sign the source code? It is important to remember that crates.io doesn't store binaries.

See also a related comment in the overall thread saying "crates.io does not host compiled artifacts. If packages on crates.io differ from their Git repository it's because of a custom pre-build step of that particular package, so a deterministic compilation toolchain won't help here."

Prove me wrong! I'm open to it.

Or be that person who is too lazy to respond with an actual comment, downvotes, and probably assumes they are right.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#64

Post by agwa »

kibwen wrote: post #56
agwa wrote: post #40

Earlier quoted context omitted.

As kibween noted above, crates.io can be changed if there's a "good reason" so it's not truly immutable either. Go does not delete modules from the module proxy except for copyright/legal reasons (I suspect crates.io would delete for these reasons too), and additionally the checksums of all modules are published in a tamper-proof transparency log ( https://sum.golang.org/ ) so if they did alter or delete a module, it…

The Go module proxy doesn't retain modules forever, as documented in the link there. And in the case where someone alters the underlying git repo by attaching a previously-used tag to a new commit, one of the following scenarios must then result: 1. The proxy delivers the changed code while issuing a warning to the end user, which if the warning is overlooked would mean that the checksum achieved nothing. 2. The prox…

In general, the Go module proxy retains modules forever, for the explicit purpose of not breaking builds (https://sum.golang.org/#faq-retract-version). crates.io can delete content also. I don't think there is much difference between crates.io and the Go module proxy in this regard - they both aim to keep source code forever to avoid breaking builds, but will delete content if there's a good reason.

If the Git tag changes, the proxy returns the original code. There is no warning that the Git tag has changed, but it can't reliably detect this anyways because the Git repository could be returning different content to different clients. I don't think there is much difference between crates.io and the Go module proxy in this regard - in neither ecosystem can you assume the Git repo matches what the packaging tool downloads. (I pointed this out here: https://news.ycombinator.com/item?id=40699948)

Where Go is different is that it provides assurance that the module proxy is providing the same code to everyone, eliminating the module proxy as a potential source of compromise. It also ensures that people who disable the module proxy and fetch code directly using the go command get the same code that the module proxy has. To reiterate, this does not help with doing code audits of Git repos - you have to either audit the code in the module proxy, or compute the checksum of the Git repo to make sure it matches the sumdb.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#65

Post by kibwen »

pabs3 wrote: post #62
kibwen wrote: post #26

Earlier quoted context omitted.

It's possible that crates.io might attempt to build a crate when published as a sort of sanity check (I don't know if this is true, but it's certainly feasible), but it doesn't distribute binaries, it distributes source code.

> it doesn't distribute binaries, it distributes source code. It definitely does contain generated files, at least one crate has Rust code generated by a Python script that is not in the crate, only in the upstream Git repository.

Yes, let's clarify: crates.io expects a Rust crate, which itself can contain whatever junk the uploader wants. But crates.io isn't taking your source, building it, and then distributing those executables; at the end of the day it's distributing the source code of a Rust crate as given by whoever published it.
Post reply on HN