Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go.sum is not a lockfile

words.filippo.io

Post reply on HN
Print view View original

41–50 of 81 posts

Re: Go.sum is not a lockfile

#41

Post by wereHamster »

JetSetIlly wrote: post #23
wereHamster wrote: post #18

A lock file, in my world, contains a cryptographic hash of dependencies. go.mod does not, it only lists tags, which are (in git) movable references. If go.sum has "no observable effect on builds", you don't know what you're building and go can download and run unverified code. I'm not a go developer and must be misunderstanding something...

By default, all go mod downloads go through the golang proxy ( https://proxy.golang.org/ ). That is part of the verification process.

Let's assume I publish a github repo with some go code, and tag a particular commit with tag v1.0.0. People start using it and put v1.0.0 into their go.mod file. They use the golang proxy to fetch the code (and that proxy does the "verification", according to your comment). Now I delete the v1.0.0 tag and re-create the tag to point to different (malicious) commit. Will the golang proxy notice? How does it verify that the people that expect the former commit under the v1.0.0 tag will actually get that and not the other (malicious) commit?

Re: Go.sum is not a lockfile

#42

Post by compsciphd »

wereHamster wrote: post #41
JetSetIlly wrote: post #23

Earlier quoted context omitted.

By default, all go mod downloads go through the golang proxy ( https://proxy.golang.org/ ). That is part of the verification process.

Let's assume I publish a github repo with some go code, and tag a particular commit with tag v1.0.0. People start using it and put v1.0.0 into their go.mod file. They use the golang proxy to fetch the code (and that proxy does the "verification", according to your comment). Now I delete the v1.0.0 tag and re-create the tag to point to different (malicious) commit. Will the golang proxy notice? How does it verify that…

yes.

From my understanding

its stored forever in the proxy cache and your new tag will never be fetched by users who go through the language's centralized infrastructure (i.e. proxy).

go can also validate the checksums (go.sum) against the languages central infrastructure that associates version->checksums.

i.e. if you cut a release, realize you made a mistake and try to fix it quitely, no user will ever see it if even one user saw the previous version (and that one user is probably you, as you probably fetched it through the proxy to see the mistake)

Re: Go.sum is not a lockfile

#43

Post by kibwen »

compsciphd wrote: post #42
wereHamster wrote: post #41

Earlier quoted context omitted.

Let's assume I publish a github repo with some go code, and tag a particular commit with tag v1.0.0. People start using it and put v1.0.0 into their go.mod file. They use the golang proxy to fetch the code (and that proxy does the "verification", according to your comment). Now I delete the v1.0.0 tag and re-create the tag to point to different (malicious) commit. Will the golang proxy notice? How does it verify that…

yes. From my understanding its stored forever in the proxy cache and your new tag will never be fetched by users who go through the language's centralized infrastructure (i.e. proxy). go can also validate the checksums (go.sum) against the languages central infrastructure that associates version->checksums. i.e. if you cut a release, realize you made a mistake and try to fix it quitely, no user will ever see it if ev…

> its stored forever in the proxy cache

This is mistaken. The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module. From the outside, we would expect that their policy is to only ever delete checksums for modules that haven't been fetched in a long time. But in general, you should not base your security model on the notion that these checksums are stored permanently.

Re: Go.sum is not a lockfile

#44

Post by onionisafruit »

nasretdinov wrote: post #17
Groxx wrote: post #4

> If the main module imports example.com/mod1/pkg1 and a separate example.com/mod1/pkg2 imports example.com/mod2, there is no way for example.com/mod2 to affect the build or run code on the developer’s machine, so you don’t need to consider it a dependency. iirc (I do not have a test setup at the moment to verify) it does affect your dependency resolution, and therefore your build, though its code does not exist in y…

These libraries are honestly so bad that whenever I have to interact with them I just split those dependencies into a separate binary :)

I dabbled in Kubernetes at work a few years ago. I was shocked at the state of their go.mod files. Especially because I had heard Kubernetes used as an example of great Go software. Right off the bat I needed to copy a dozen or so `replace` statements into my go.mod. I was sure I just didn’t understand something and spent an hour or so looking for the “right way” only to discover other open source controllers that had the same replaces in their go.mods.

Maybe that was a transitory stage and it’s straightened out now. I certainly hope so.

Re: Go.sum is not a lockfile

#45

Post by kibwen »

alias_neo wrote: post #22
inglor wrote: post #21

Earlier quoted context omitted.

You are not misunderstanding anything, I use Go and Rust/TypeScript in my daily work and you are correct - it is the OP that does not understand why people use lockfiles in CI (to prevent minor updates and changes in upstream through verifying a hash signature).

I would hazard a guess that the (former) head of the Go security team at Google (OP) _does_ in fact understand.

They may be an expert in Go, but from their writing they appear to be misunderstanding (or at least misrepresenting) how things work in other languages. See the previous discussion here: https://lobste.rs/s/exv2eq/go_sum_is_not_lockfile

Re: Go.sum is not a lockfile

#46

Post by agwa »

kibwen wrote: post #43
compsciphd wrote: post #42

Earlier quoted context omitted.

yes. From my understanding its stored forever in the proxy cache and your new tag will never be fetched by users who go through the language's centralized infrastructure (i.e. proxy). go can also validate the checksums (go.sum) against the languages central infrastructure that associates version->checksums. i.e. if you cut a release, realize you made a mistake and try to fix it quitely, no user will ever see it if ev…

> its stored forever in the proxy cache This is mistaken. The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module. From the outside, we would expect that their policy is to only ever delete checksums for modules that haven't been fetched in a long time. But in general, you should not base your security model on the notion that these checksums are stored permanen…

> The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module

Incorrect. Checksums are stored forever, in a Merkle Tree, meaning if the proxy were to ever delete a checksum, it would be detected (and yes, people like me are checking - https://sourcespotter.com/sumdb).

Like any code host, the proxy does not guarantee that the code for a module will be available forever, since code may have to be removed for legal reasons.

But you absolutely can rely on the checksum being preserved and thus you can be sure you'll never be given different code for a particular version.

Re: Go.sum is not a lockfile

#47

Post by terinjokes »

jchw wrote: post #37
skywhopper wrote: post #32

Earlier quoted context omitted.

But a “version” as listed in go.mod may have different hashes over time, if tags are changed. That’s the issue.

This doesn't make any sense to me. If you wanted to verify the contents of a dependency, you would want to check go.sum. That's what it is there for, after all. So if you wanted to fetch the dependencies, then you would want to use it to verify hashes. If all you care about the is the versions of dependencies, you really can (and should) trust go.mod alone. You can do this because there are multiple overlapping mecha…

A local cache of sums are also stored in (iirc) $GOCACHE, so even if you delete go.sum from the project, the local toolchain should still be able to verify module versions previously seen without needing to call out to the Sum DB.

Re: Go.sum is not a lockfile

#48

Post by kibwen »

agwa wrote: post #46
kibwen wrote: post #43

Earlier quoted context omitted.

> its stored forever in the proxy cache This is mistaken. The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module. From the outside, we would expect that their policy is to only ever delete checksums for modules that haven't been fetched in a long time. But in general, you should not base your security model on the notion that these checksums are stored permanen…

> The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module Incorrect. Checksums are stored forever, in a Merkle Tree, meaning if the proxy were to ever delete a checksum, it would be detected (and yes, people like me are checking - https://sourcespotter.com/sumdb ). Like any code host, the proxy does not guarantee that the code for a module will be available fore…

Ah, my mistake. I had read in the FAQ that it does not guarantee that data is stored forever, but overlooked the part about preserving checksums specifically.

Re: Go.sum is not a lockfile

#49

Post by onionisafruit »

ncruces wrote: post #40
skywhopper wrote: post #32

Earlier quoted context omitted.

But a “version” as listed in go.mod may have different hashes over time, if tags are changed. That’s the issue.

Conversely, I can say that an hash being in go.sum doesn't mean it will be used for anything. Only that if the corresponding version does get used, and the hash doesn't match, you get an error. But you can have multiple versions of the same dep in your go.sum - or none at all - and this has no bearing on what version gets picked when you build your module. The version that does get picked is the one in go.mod of the…

Although he doesn't spell it out, I suspect this is the primary misunderstanding that drove Filo to open with "I need everyone to stop looking at go.sum, especially to analyze dependency graphs". I've had more than one code reviewer ding me on a module showing up in go.sum. Usually it's a situation where a dependency has tests for compatibility with some other module so that other module gets added to go.sum. Given Filo is a professional open source maintainer, any annoyance I've run into he's probably experienced 100x.

Re: Go.sum is not a lockfile

#50

Post by onionisafruit »

jchw wrote: post #37
skywhopper wrote: post #32

Earlier quoted context omitted.

But a “version” as listed in go.mod may have different hashes over time, if tags are changed. That’s the issue.

This doesn't make any sense to me. If you wanted to verify the contents of a dependency, you would want to check go.sum. That's what it is there for, after all. So if you wanted to fetch the dependencies, then you would want to use it to verify hashes. If all you care about the is the versions of dependencies, you really can (and should) trust go.mod alone. You can do this because there are multiple overlapping mecha…

> If you wanted to verify the contents of a dependency, you would want to check go.sum

You're right, but also TFA says "There is truly no use case for ever parsing it outside of cmd/go". Since cmd/go verifies the contents of your dependencies, the point generally stands. If you don't trust cmd/go to verify a dependency, then you have a valid exception to the rule.

Post reply on HN