Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I've compared nearly all Rust crates.io crates to contents of their Git repos

mastodon.social

Post reply on HN
Print view View original

31–40 of 65 posts

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#31

Post by mmastrac »

thayne wrote: post #18
jonahx wrote: post #4

Earlier quoted context omitted.

Not a Rust dev so maybe a dumb question, but is this more involved than just running diffs? If so, what needs to be done?

Most of the diffs are probably innocuous. I suspect the most common diff would be the version line of Cargo.toml, both from CI that automatically updates that line, and people who forgot to update it before making a tag in git.

As someone with a crate that's in the 50MM plus range, this happens all the time. I really should automate this via a GH action.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#32

Post by dgoldstein0 »

thayne wrote: post #24
corytheboyd wrote: post #8

How crazy would it be to have a package repository that also builds the artifacts it distributes? You’d need a high barrier to entry to save on costs and time sifting through garbage. Perhaps it’s this high barrier that would prevent such a repository from taking off though. Perhaps this is just a really dumb step on a path leading back to simple checksum validations… though with those, you’re only validating that wh…

go kind of solves that by making the git repo the source of truth for a package, and host a cache for it. The problem with it is you need the full git url in every file you import it. which is a pain if the repo changes locations, or you want to use a fork or a local version. Versioning is also tricky, to the point that go recommends creating a separate branch for a major/breaking version, which requires updating eve…

bazel has modules now: https://bazel.build/external/module

Not tried them but they look like a reasonable dep handling solution on paper - each module can declare it's own dependencies and bazel will figure it out for you like a package manager. Their old workspaces way of doing it was a nightmare, as while patterns emerged where repos would export a function to register their dependencies, the first declaration of any name would win and thus you weren't guaranteed to have a compatible set of workspaces at the end.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#33

Post by moffkalast »

sosodev wrote: post #28
kibwen wrote: post #22

Earlier quoted context omitted.

I'm not sure what is meant by "downloading from Git", I assume you mean downloading from Github. And Github is far less secure than what crates.io does, because crates.io is immutable (once published, uploaders can't change anything without opening a support ticket which will get rejected if they don't have a good reason), whereas Github history is trivially rewriteable. This means that if you rely on "v1.2.3" of a l…

Go modules can be hosted in any Git repository. The Go toolchain also keeps hashes of the selected tag so if you've reviewed it once it will never change without you explicitly giving it the ok.

> giving it the ok

You mean giving it the Go ahead? ;)

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#34

Post by estebank »

timeon wrote: post #29
makeworld wrote: post #20

This is why I like what Go does, where you're downloading from Git directly (optionally proxied through Google, yes)

You can do that with Rust as well if you define path of dependency to git repo (or local dir).

Be aware that you cannot publish on crates.io if you do that: either you buy into the system (so that you can ensure that you can rebuild in perpetuity) or not at all (so you end up with a crate that can depenend partially on crates.io, but must always be consumed directly from a repo or directory.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#35

Post by aseipp »

kibwen wrote: post #22
makeworld wrote: post #20

This is why I like what Go does, where you're downloading from Git directly (optionally proxied through Google, yes)

I'm not sure what is meant by "downloading from Git", I assume you mean downloading from Github. And Github is far less secure than what crates.io does, because crates.io is immutable (once published, uploaders can't change anything without opening a support ticket which will get rejected if they don't have a good reason), whereas Github history is trivially rewriteable. This means that if you rely on "v1.2.3" of a l…

> conversely, relying on a git tag of "v1.2.3" from a random Github repo could be anything at any point.

I don't know of a single modern build tool that can do this but doesn't require or record this information specifically? Maybe the earlier versions of Go? (I know they've gone through a few changes in module/import strategies.)

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#36

Post by estebank »

fsmv wrote: post #27
kibwen wrote: post #22

Earlier quoted context omitted.

I'm not sure what is meant by "downloading from Git", I assume you mean downloading from Github. And Github is far less secure than what crates.io does, because crates.io is immutable (once published, uploaders can't change anything without opening a support ticket which will get rejected if they don't have a good reason), whereas Github history is trivially rewriteable. This means that if you rely on "v1.2.3" of a l…

The goproxy server makes it somewhat immutable for go too. Once they have a version cached they will never delete it. You can only supercede it with a new version and mark the old version as bad.

TBH, "somewhat immutable" is not "immutable". The Go approach aids to limit the effects of an attempted attach where you're misled into building your project with different code than originally intended, but does nothing to guarantee continuity over time of dependencies being available. For that you have to rely on vendoring.

The crates.io approach instead defends you both from a dependency changing silently and from it disappearing from one day to the next, without having to deal with vendoring.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#38

Post by agwa »

makeworld wrote: post #20

This is why I like what Go does, where you're downloading from Git directly (optionally proxied through Google, yes)

The problem is that if you clone the Git repository, or view it on GitHub, you have no assurance that you're seeing the same code that the go command or the Go module proxy saw. The author of a malicious module could change the Git tag to point to a different, benign, commit after the Go module proxy stores the malicious copy. There are other tricks an attacker can play as well: https://github.com/golang/go/issues/66653

Ultimately, if you're doing a code audit, you have to compute the checksum of the code that you're looking at, and compare it against the entry in go.sum or the checksum database to make sure you're auditing the right copy.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#39

Post by SubiculumCode »

SubiculumCode wrote: post #3
jraph wrote: post #2

Good initiative. Now people need to go through this and do the reviews :-) Next step would be to do reproducible builds (if it's not already the case).

First pass gpt?

Heavily down voted, which is fair because I didn't really explain what I meant, which was: Would using LLM's to parse the generated diffs, as a first pass, be useful/efficient for spotting and interpreting discrepancies?

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#40

Post by agwa »

estebank wrote: post #36
fsmv wrote: post #27

Earlier quoted context omitted.

The goproxy server makes it somewhat immutable for go too. Once they have a version cached they will never delete it. You can only supercede it with a new version and mark the old version as bad.

TBH, "somewhat immutable" is not "immutable". The Go approach aids to limit the effects of an attempted attach where you're misled into building your project with different code than originally intended, but does nothing to guarantee continuity over time of dependencies being available. For that you have to rely on vendoring. The crates.io approach instead defends you both from a dependency changing silently and from…

As kibween noted above, crates.io can be changed if there's a "good reason" so it's not truly immutable either.

Go does not delete modules from the module proxy except for copyright/legal reasons (I suspect crates.io would delete for these reasons too), and additionally the checksums of all modules are published in a tamper-proof transparency log (https://sum.golang.org/) so if they did alter or delete a module, it would be detected.

Post reply on HN