Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go.sum is not a lockfile

words.filippo.io

Post reply on HN
Print view View original

71–80 of 81 posts

Re: Go.sum is not a lockfile

#71

Post by quacker »

nvarsj wrote: post #31
Groxx wrote: post #4

> If the main module imports example.com/mod1/pkg1 and a separate example.com/mod1/pkg2 imports example.com/mod2, there is no way for example.com/mod2 to affect the build or run code on the developer’s machine, so you don’t need to consider it a dependency. iirc (I do not have a test setup at the moment to verify) it does affect your dependency resolution, and therefore your build, though its code does not exist in y…

godep and vendoring completely solved this problem. Go modules relies too heavily on dependencies being good citizens, which is a very naive approach to dependency management.

You can still do vendoring with Go modules, using the `go mod vendor` command.

Re: Go.sum is not a lockfile

#72

Post by typical182 »

kibwen wrote: post #45
alias_neo wrote: post #22

Earlier quoted context omitted.

I would hazard a guess that the (former) head of the Go security team at Google (OP) _does_ in fact understand.

They may be an expert in Go, but from their writing they appear to be misunderstanding (or at least misrepresenting) how things work in other languages. See the previous discussion here: https://lobste.rs/s/exv2eq/go_sum_is_not_lockfile

> They may be an expert in Go, but from their writing they appear to be misunderstanding (or at least misrepresenting) how things work in other languages

Thanks for that link.

Based on reading through that whole discussion there just now and my understanding of the different ecosystems, my conclusion is that certainly people there are telling Filippo Valsorda that he is misunderstanding how things work in other languages, but then AFAICT Filippo or others chime in to explain how he is in fact not misunderstanding.

This subthread to me was a seemingly prototypical exchange there:

https://lobste.rs/s/exv2eq/go_sum_is_not_lockfile#c_d26oq4

Someone in that subthread tells Filippo (FiloSottile) that he is misunderstanding cargo behavior, but Filippo then reiterates which behavior he is talking about (add vs. install), Filippo does a simple test to illustrate his point, and some others seem to agree that he is correct in what he originally said.

That said, YMMV, and that overall discussion does certainly seem to have some confusion and people seemingly talking past each other (e.g., some people mixing up "dependents" vs. "dependencies", etc.).

Re: Go.sum is not a lockfile

#73

Post by nvarsj »

quacker wrote: post #71
nvarsj wrote: post #31

Earlier quoted context omitted.

godep and vendoring completely solved this problem. Go modules relies too heavily on dependencies being good citizens, which is a very naive approach to dependency management.

You can still do vendoring with Go modules, using the `go mod vendor` command.

I'm a bit out of date now, but 'go mod vendor' did not in fact vendor all dependencies when I last used it. It seemed more like a CI caching mechanism then actual complete dependency vendoring.

Re: Go.sum is not a lockfile

#74

Post by typical182 »

compsciphd wrote: post #69
kibwen wrote: post #43

Earlier quoted context omitted.

> its stored forever in the proxy cache This is mistaken. The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module. From the outside, we would expect that their policy is to only ever delete checksums for modules that haven't been fetched in a long time. But in general, you should not base your security model on the notion that these checksums are stored permanen…

ok, I guess I was wrong about the cache, but not the checksums. I was somewhat under the impression that it was forever due to the getting rid of vendoring. Getting rid of vendoring (to me) only makes sense if its cached forever (otherwise vendoring has significant value).

Go modules did not get rid of vendoring. You can do 'go mod vendor' and have been able to do so since Go modules were first introduced.

How long the google-run module cache (aka, module proxy or module mirror) at https://proxy.golang.org caches the contents of modules is I think slightly nuanced.

That page includes:

> Whenever possible, the mirror aims to cache content in order to avoid breaking builds for people that depend on your package

But that page also discusses how modules might need to be removed for legal reasons or if a module does not have a known Open Source license:

> proxy.golang.org does not save all modules forever. There are a number of reasons for this, but one reason is if proxy.golang.org is not able to detect a suitable license. In this case, only a temporarily cached copy of the module will be made available, and may become unavailable if it is removed from the original source and becomes outdated.

If interested, there's a good overview of how it all works in one of the older official announcement blog posts (in particular, the "Module Index", "Module Authentication", "Module Mirrors" sections there):

https://go.dev/blog/modules2019#module-index

Re: Go.sum is not a lockfile

#75

Post by kibwen »

typical182 wrote: post #72
kibwen wrote: post #45

Earlier quoted context omitted.

They may be an expert in Go, but from their writing they appear to be misunderstanding (or at least misrepresenting) how things work in other languages. See the previous discussion here: https://lobste.rs/s/exv2eq/go_sum_is_not_lockfile

> They may be an expert in Go, but from their writing they appear to be misunderstanding (or at least misrepresenting) how things work in other languages Thanks for that link. Based on reading through that whole discussion there just now and my understanding of the different ecosystems, my conclusion is that certainly people there are telling Filippo Valsorda that he is misunderstanding how things work in other langu…

> but then AFAICT Filippo or others chime in to explain how he is in fact not misunderstanding.

I don't get this impression. Rather, as you say, I get the impression that people are talking past each other, a property which also extends to the author, and the overall failure to reach a mutual understanding of terms only contributes to muddying the waters all around. Here's a direct example that's still in the OP:

"The lockfile (e.g. uv.lock, package-lock.json, Cargo.lock) is a relatively recent innovation in some ecosystems, and it lists the actual versions used in the most recent build. It is not really human-readable, and is ignored by dependents, allowing the rapid spread of supply-chain attacks."

At the end there, what the author is talking about has nothing to do with lockfiles specifically, let alone when they are applied or ignored, but rather to do with the difference between minimum-version selection (which Go uses) and max-compatible-version selection.

Here's another one:

"In other ecosystems, package resolution time going down below 1s is celebrated"

This is repeating the mistaken claims that Russ Cox made years ago when he designed Go's current packaging system. Package resolution in e.g. Cargo is almost too fast to measure, even on large dependency trees.

Re: Go.sum is not a lockfile

#76

Post by majewsky »

nvarsj wrote: post #73
quacker wrote: post #71

Earlier quoted context omitted.

You can still do vendoring with Go modules, using the `go mod vendor` command.

I'm a bit out of date now, but 'go mod vendor' did not in fact vendor all dependencies when I last used it. It seemed more like a CI caching mechanism then actual complete dependency vendoring.

This is not the experience that I have. I use vendoring heavily because it's nice to be able to review all the relevant code changes when Renovatebot updates a dependency [1], and to get a feel for how heavy or lightweight certain dependencies are. If vendoring was incomplete, I would see it trying to download missing dependencies at compile time, but the "go: downloading $MODULE $VERSION" lines only show up when I expect them too, e.g. during "go get -u" or "go mod tidy/vendor".

[1] Before you ask, I'm not reading the full diff on something like x/sys. Mostly on third-party dependencies where I find it harder to judge the reliability of the maintainers.

Re: Go.sum is not a lockfile

#77

Post by majewsky »

peterldowns wrote: post #3

Correct, but it's been ages and the default actions/setup-go github action still uses go.sum instead. I see that someone already commented on the longstanding issue to reference this post, and that there is some hope that they'll update it! https://github.com/actions/setup-go/issues/478

At this point I'm just assuming that all of the official actions under https://github.com/actions are unmaintained garbage until proven otherwise, given that high-impact issues like this can sit for years without as much as an initial triage: https://github.com/actions/delete-package-versions/issues/16...

Re: Go.sum is not a lockfile

#78

Post by jamesfinlayson »

kbolino wrote: post #58
supriyo-biswas wrote: post #25

Earlier quoted context omitted.

I just write my own, small, focused client for these AWS services since the SDKs are generally just so unergonomic. With vibe coding becoming a thing it's become even easier to do that.

You may already know these things, but for others who may not: the AWS SDK for Go suffers from shockingly bad discoverability. For example, take the S3 library, github.com/aws/aws-sdk-go-v2/service/s3. If you have an s3.Client and you look at e.g. the ListObjectsV2 method, you might have no idea that there is a ListObjectsV2Paginator which makes it much easier to use, because nowhere in the method docs is it mentione…

Yeah the AWS PHP SDK is similarly bad. The Java SDK is okay but still completely lacking in examples for anything outside of the basics.

AWS seems to optimise for an SDK that be completely generated but not for an SDK that tells you what you want to know.

Re: Go.sum is not a lockfile

#79

Post by compsciphd »

typical182 wrote: post #74
compsciphd wrote: post #69

Earlier quoted context omitted.

ok, I guess I was wrong about the cache, but not the checksums. I was somewhat under the impression that it was forever due to the getting rid of vendoring. Getting rid of vendoring (to me) only makes sense if its cached forever (otherwise vendoring has significant value).

Go modules did not get rid of vendoring. You can do 'go mod vendor' and have been able to do so since Go modules were first introduced. How long the google-run module cache (aka, module proxy or module mirror) at https://proxy.golang.org caches the contents of modules is I think slightly nuanced. That page includes: > Whenever possible, the mirror aims to cache content in order to avoid breaking builds for people tha…

ok, 1) so would it be fair to modify my statement that it basically tries to cache forever unless its can't determine that its legally allowed to cache forever?

2) you're right, glanced at kubernetes (been a long time since I worked on it) and they still have a vendor directory that gets updated regularly.

Re: Go.sum is not a lockfile

#80

Post by jchw »

onionisafruit wrote: post #50
jchw wrote: post #37

Earlier quoted context omitted.

This doesn't make any sense to me. If you wanted to verify the contents of a dependency, you would want to check go.sum. That's what it is there for, after all. So if you wanted to fetch the dependencies, then you would want to use it to verify hashes. If all you care about the is the versions of dependencies, you really can (and should) trust go.mod alone. You can do this because there are multiple overlapping mecha…

> If you wanted to verify the contents of a dependency, you would want to check go.sum You're right, but also TFA says "There is truly no use case for ever parsing it outside of cmd/go". Since cmd/go verifies the contents of your dependencies, the point generally stands. If you don't trust cmd/go to verify a dependency, then you have a valid exception to the rule.

Agreed. Arguably, though, it would be much more reasonable to trust cmd/go to verify a dependency than it would to trust your own code. A lot more effort is put into it and it has a proper security process established. So I think the point is, if you find yourself actually needing to verify the go.sum, not by using cmd/go, you are very likely doing something wrong.
Post reply on HN