Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go.sum is not a lockfile

words.filippo.io

Post reply on HN
Print view View original

61–70 of 81 posts

Re: Go.sum is not a lockfile

#61

Post by ncruces »

wereHamster wrote: post #59
agwa wrote: post #46

Earlier quoted context omitted.

> The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module Incorrect. Checksums are stored forever, in a Merkle Tree, meaning if the proxy were to ever delete a checksum, it would be detected (and yes, people like me are checking - https://sourcespotter.com/sumdb ). Like any code host, the proxy does not guarantee that the code for a module will be available fore…

Ok. So to answer the question whether the code for v1.0.0 that I downloaded today is the same as I downloaded yesterday (or whether the code that I get is the same as the one my coworker is getting) you basically have to trust Google.

Not really.

For the question “is the data in the checksum database immutable” you can trust people like the parent, who double checks what Google is doing.

For the question “is it the same data that can be downloaded directly from the repos” you can skip the proxy to download dependencies, then do it again with the proxy, and compare.

So I'd say you don't need to trust Google at all in this case.

Re: Go.sum is not a lockfile

#62

Post by [deleted] »

wereHamster wrote: post #59
agwa wrote: post #46

Earlier quoted context omitted.

> The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module Incorrect. Checksums are stored forever, in a Merkle Tree, meaning if the proxy were to ever delete a checksum, it would be detected (and yes, people like me are checking - https://sourcespotter.com/sumdb ). Like any code host, the proxy does not guarantee that the code for a module will be available fore…

Ok. So to answer the question whether the code for v1.0.0 that I downloaded today is the same as I downloaded yesterday (or whether the code that I get is the same as the one my coworker is getting) you basically have to trust Google.

[deleted]

Re: Go.sum is not a lockfile

#63

Post by agwa »

wereHamster wrote: post #59
agwa wrote: post #46

Earlier quoted context omitted.

> The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module Incorrect. Checksums are stored forever, in a Merkle Tree, meaning if the proxy were to ever delete a checksum, it would be detected (and yes, people like me are checking - https://sourcespotter.com/sumdb ). Like any code host, the proxy does not guarantee that the code for a module will be available fore…

Ok. So to answer the question whether the code for v1.0.0 that I downloaded today is the same as I downloaded yesterday (or whether the code that I get is the same as the one my coworker is getting) you basically have to trust Google.

The checksums are published in a transparency log, which uses a Merkle Tree[1] to make the attack you describe detectable. Source Spotter, which is unaffiliated with Google, continuously verifies that the log contains only one checksum per module version.

If Google were to present you with a different view of the Merkle Tree with different checksums in it, they'd have to forever show you, and only you, that view. If they accidentally show someone else that view, or show you the real view, the go command would detect it. This will eventually be strengthened further with witnessing[2], which will ensure that everyone's view of the log is the same. In the meantime, you / your coworker can upload your view of the log (in $GOPATH/pkg/sumdb/sum.golang.org/latest) to Source Spotter and it will tell you if it's consistent with its view:

  $ curl --data-binary "@$(go env GOPATH)/pkg/sumdb/sum.golang.org/latest" https://gossip.api.sourcespotter.com/sum.golang.org 
  consistent: this STH is consistent with other STHs that we've seen from sum.golang.org
[1] https://research.swtch.com/tlog

[2] https://github.com/C2SP/C2SP/blob/main/tlog-witness.md

Re: Go.sum is not a lockfile

#64

Post by kbolino »

agwa wrote: post #46
kibwen wrote: post #43

Earlier quoted context omitted.

> its stored forever in the proxy cache This is mistaken. The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module. From the outside, we would expect that their policy is to only ever delete checksums for modules that haven't been fetched in a long time. But in general, you should not base your security model on the notion that these checksums are stored permanen…

> The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module Incorrect. Checksums are stored forever, in a Merkle Tree, meaning if the proxy were to ever delete a checksum, it would be detected (and yes, people like me are checking - https://sourcespotter.com/sumdb ). Like any code host, the proxy does not guarantee that the code for a module will be available fore…

Here's another person auditing the checksum database: https://raphting.dev/posts/gosumdb-live-again/

Re: Go.sum is not a lockfile

#65

Post by [deleted] »

LukeShu wrote: post #8
djha-skin wrote: post #6

> Instead, just look at go.mod. It lists the precise version at which all dependencies are built. No, it does not. Minimum version selection means that the libraries will at least be that version, but it could be substituted for a later version if a transient dependency asks for such. That I'm reading this blog post at all suggests there is a "market" for a single checksum/version manifest, which data is currently ho…

The correct answer is `go mod vendor && cat vendor/modules.txt`

[deleted]

Re: Go.sum is not a lockfile

#66

Post by nullpoint420 »

jerf wrote: post #55
onionisafruit wrote: post #44

Earlier quoted context omitted.

I dabbled in Kubernetes at work a few years ago. I was shocked at the state of their go.mod files. Especially because I had heard Kubernetes used as an example of great Go software. Right off the bat I needed to copy a dozen or so `replace` statements into my go.mod. I was sure I just didn’t understand something and spent an hour or so looking for the “right way” only to discover other open source controllers that ha…

I think a lot of people assume Kubernetes must be a good Go example, because it's so big and successful. But it started life as a Java project and was ported into Go, and that shows in some of the architecture, Also, in general across all languages, picking the absolute largest projects you can is often not a great idea in terms of copying design unless you too are going to be that large, e.g., I wouldn't suggest usi…

I like your comparison about Firefox not being a great example of "good" C/C++.

I'd personally consider myself comfortable in C/C++. I've built Wayland compositors, H264 backends for live-streaming, and built Chromium occasionally for testing. Despite being a die-hard Firefox (zen) user - I still have not been able to compile Firefox! To be fair, this was pre-Firefox quantum days so I hope their SVM and build tools have improved.

Re: Go.sum is not a lockfile

#67

Post by lokar »

jerf wrote: post #55
onionisafruit wrote: post #44

Earlier quoted context omitted.

I dabbled in Kubernetes at work a few years ago. I was shocked at the state of their go.mod files. Especially because I had heard Kubernetes used as an example of great Go software. Right off the bat I needed to copy a dozen or so `replace` statements into my go.mod. I was sure I just didn’t understand something and spent an hour or so looking for the “right way” only to discover other open source controllers that ha…

I think a lot of people assume Kubernetes must be a good Go example, because it's so big and successful. But it started life as a Java project and was ported into Go, and that shows in some of the architecture, Also, in general across all languages, picking the absolute largest projects you can is often not a great idea in terms of copying design unless you too are going to be that large, e.g., I wouldn't suggest usi…

I remember early on the Go people complaining about the K8S Go code

Re: Go.sum is not a lockfile

#68

Post by lokar »

jchw wrote: post #37
skywhopper wrote: post #32

Earlier quoted context omitted.

But a “version” as listed in go.mod may have different hashes over time, if tags are changed. That’s the issue.

This doesn't make any sense to me. If you wanted to verify the contents of a dependency, you would want to check go.sum. That's what it is there for, after all. So if you wanted to fetch the dependencies, then you would want to use it to verify hashes. If all you care about the is the versions of dependencies, you really can (and should) trust go.mod alone. You can do this because there are multiple overlapping mecha…

Probably unpopular, but I just use Bazel and pick the versions of software I use.

I know the current attitude is to just blindly trust 3rd party libraries (current and all future versions) and all of their dependencies, but I just can't accept that. This is just unsustainable.

I guess I'm old or something.

Re: Go.sum is not a lockfile

#69

Post by compsciphd »

kibwen wrote: post #43
compsciphd wrote: post #42

Earlier quoted context omitted.

yes. From my understanding its stored forever in the proxy cache and your new tag will never be fetched by users who go through the language's centralized infrastructure (i.e. proxy). go can also validate the checksums (go.sum) against the languages central infrastructure that associates version->checksums. i.e. if you cut a release, realize you made a mistake and try to fix it quitely, no user will ever see it if ev…

> its stored forever in the proxy cache This is mistaken. The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module. From the outside, we would expect that their policy is to only ever delete checksums for modules that haven't been fetched in a long time. But in general, you should not base your security model on the notion that these checksums are stored permanen…

ok, I guess I was wrong about the cache, but not the checksums. I was somewhat under the impression that it was forever due to the getting rid of vendoring. Getting rid of vendoring (to me) only makes sense if its cached forever (otherwise vendoring has significant value).

Re: Go.sum is not a lockfile

#70

Post by jchw »

lokar wrote: post #68
jchw wrote: post #37

Earlier quoted context omitted.

This doesn't make any sense to me. If you wanted to verify the contents of a dependency, you would want to check go.sum. That's what it is there for, after all. So if you wanted to fetch the dependencies, then you would want to use it to verify hashes. If all you care about the is the versions of dependencies, you really can (and should) trust go.mod alone. You can do this because there are multiple overlapping mecha…

Probably unpopular, but I just use Bazel and pick the versions of software I use. I know the current attitude is to just blindly trust 3rd party libraries (current and all future versions) and all of their dependencies, but I just can't accept that. This is just unsustainable. I guess I'm old or something.

Go MVS does not require you to blindly trust 3rd party libraries. Certainly not "current and all future versions". Go modules also offer hermetic and reproducible dependency resolution by default.
Post reply on HN