Live data from Hacker News

I've compared nearly all Rust crates.io crates to contents of their Git repos

mastodon.social

21–30 of 65 posts

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#21

This is why I like what Go does, where you're downloading from Git directly (optionally proxied through Google, yes)

Related: Proxying can be disabled by setting the environment variable GOPROXY=direct [0]. I put it in my bashrc.

[0] https://www.practical-go-lessons.com/chap-18-go-module-proxi...

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#22

This is why I like what Go does, where you're downloading from Git directly (optionally proxied through Google, yes)

I'm not sure what is meant by "downloading from Git", I assume you mean downloading from Github. And Github is far less secure than what crates.io does, because crates.io is immutable (once published, uploaders can't change anything without opening a support ticket which will get rejected if they don't have a good reason), whereas Github history is trivially rewriteable. This means that if you rely on "v1.2.3" of a library from crates.io, that's always going to give everyone the same code; conversely, relying on a git tag of "v1.2.3" from a random Github repo could be anything at any point.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#23
post #6

How could you rank them for review priority? Use a combination of repo popularity multiplied by amount of significant differences? Where significant differences are determined by excluding non-code files?

Looking through the code, it already ignores the majority of non-code files

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#24

How crazy would it be to have a package repository that also builds the artifacts it distributes? You’d need a high barrier to entry to save on costs and time sifting through garbage. Perhaps it’s this high barrier that would prevent such a repository from taking off though. Perhaps this is just a really dumb step on a path leading back to simple checksum validations… though with those, you’re only validating that wh…

go kind of solves that by making the git repo the source of truth for a package, and host a cache for it.

The problem with it is you need the full git url in every file you import it. which is a pain if the repo changes locations, or you want to use a fork or a local version. Versioning is also tricky, to the point that go recommends creating a separate branch for a major/breaking version, which requires updating every import statement.

I think a good middle ground would be to have a central repository and/or package configuration file that maps package names to git repos and versions to commits (possibly via tags). And of course use hashes to lock the version to specific contents.

Bazel kind of does this, but it doesn't have any built in version resolution or transitive dependency resolution (although in some cases there are other tools that help). And it can add a lot of complexity that you may not need.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#25
post #12
post #7

Deterministic compilation is the best way to let people validate what they are downloading from repositories is what is in the codebases.

crates.io does not host compiled artifacts. If packages on crates.io differ from their Git repository it's because of a custom pre-build step of that particular package, so a deterministic compilation toolchain won't help here.

There are other possible reasons for them not matching:

- files not being tracked in the repo

- files being part of the repo not being part of the published crate

- publishing with allow dirty from a local copy of the repo with changes that haven't been committed

- publishing from a commit that hasn't been pushed

I'm sure there are more.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#26

How crazy would it be to have a package repository that also builds the artifacts it distributes? You’d need a high barrier to entry to save on costs and time sifting through garbage. Perhaps it’s this high barrier that would prevent such a repository from taking off though. Perhaps this is just a really dumb step on a path leading back to simple checksum validations… though with those, you’re only validating that wh…

crates.io already builds the artefacts. But the code-source that is sent to crates.io is not necessarily the same as the one in the public repo linked to the crate.

It's possible that crates.io might attempt to build a crate when published as a sort of sanity check (I don't know if this is true, but it's certainly feasible), but it doesn't distribute binaries, it distributes source code.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#27
post #22

This is why I like what Go does, where you're downloading from Git directly (optionally proxied through Google, yes)

I'm not sure what is meant by "downloading from Git", I assume you mean downloading from Github. And Github is far less secure than what crates.io does, because crates.io is immutable (once published, uploaders can't change anything without opening a support ticket which will get rejected if they don't have a good reason), whereas Github history is trivially rewriteable. This means that if you rely on "v1.2.3" of a l…

The goproxy server makes it somewhat immutable for go too. Once they have a version cached they will never delete it. You can only supercede it with a new version and mark the old version as bad.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#28
post #22

This is why I like what Go does, where you're downloading from Git directly (optionally proxied through Google, yes)

I'm not sure what is meant by "downloading from Git", I assume you mean downloading from Github. And Github is far less secure than what crates.io does, because crates.io is immutable (once published, uploaders can't change anything without opening a support ticket which will get rejected if they don't have a good reason), whereas Github history is trivially rewriteable. This means that if you rely on "v1.2.3" of a l…

Go modules can be hosted in any Git repository. The Go toolchain also keeps hashes of the selected tag so if you've reviewed it once it will never change without you explicitly giving it the ok.

Re: I've compared nearly all Rust crates.io crates to contents of their Git repos

#30
post #22

This is why I like what Go does, where you're downloading from Git directly (optionally proxied through Google, yes)

I'm not sure what is meant by "downloading from Git", I assume you mean downloading from Github. And Github is far less secure than what crates.io does, because crates.io is immutable (once published, uploaders can't change anything without opening a support ticket which will get rejected if they don't have a good reason), whereas Github history is trivially rewriteable. This means that if you rely on "v1.2.3" of a l…

That’s true, except that the lockfile records revision as a commit sha.

https://github.com/apex/up-examples/blob/master/oss/golang-g...

Post reply on HN