Live data from Hacker News

Go.sum is not a lockfile

words.filippo.io

31–40 of 81 posts

Re: Go.sum is not a lockfile

#31
post #4

> If the main module imports example.com/mod1/pkg1 and a separate example.com/mod1/pkg2 imports example.com/mod2, there is no way for example.com/mod2 to affect the build or run code on the developer’s machine, so you don’t need to consider it a dependency. iirc (I do not have a test setup at the moment to verify) it does affect your dependency resolution, and therefore your build, though its code does not exist in y…

godep and vendoring completely solved this problem.

Go modules relies too heavily on dependencies being good citizens, which is a very naive approach to dependency management.

Re: Go.sum is not a lockfile

#32
post #11

> Instead, just look at go.mod. It lists the precise version at which all dependencies are built. No, it does not. Minimum version selection means that the libraries will at least be that version, but it could be substituted for a later version if a transient dependency asks for such. That I'm reading this blog post at all suggests there is a "market" for a single checksum/version manifest, which data is currently ho…

> No, it does not. Minimum version selection means that the libraries will at least be that version, but it could be substituted for a later version if a transient dependency asks for such. No? All dependencies - direct and indirect - are listed in your go.mod. Your module - as is - depends on nothing else. And those exact versions will be used to build it, if yours is the main module . If your module is used as a de…

But a “version” as listed in go.mod may have different hashes over time, if tags are changed. That’s the issue.

Re: Go.sum is not a lockfile

#33
post #4

> If the main module imports example.com/mod1/pkg1 and a separate example.com/mod1/pkg2 imports example.com/mod2, there is no way for example.com/mod2 to affect the build or run code on the developer’s machine, so you don’t need to consider it a dependency. iirc (I do not have a test setup at the moment to verify) it does affect your dependency resolution, and therefore your build, though its code does not exist in y…

It's tricky, to the point that I made a little playground to explore it.

https://github.com/FiloSottile/mostly-harmless/tree/main/dep...

The example.com/mod2 go.mod does not in fact affect version resolution, because it's not even fetched. However, it affects the example.com/mod1 go.mod, and the example.com/mod1 go.mod affects version resolution.

This doesn't help with the problem you are describing, but it still has value from a security point of view, because example.com/mod2 truly doesn't matter except to the extent that was already checked into example.com/mod1, which you do need to trust.

If you try to "go build" or "go test" something in example.com/mod2, you actually do get an error since Go 1.17, as if it was not in your dependency tree at all. You need to "go get" it like any new dependency.

Re: Go.sum is not a lockfile

#34
The pyproject.toml, package.json, and Cargo.toml are declarative project configuration files. While the Rust community refers to Cargo.toml as a manifest, it is not a comprehensive and detailed list of a build. That is the lock file.

While go.mod does not allow for explicit version ranges, the versions given are the minimum versions. In other words, the versions given are the lower bound of the compatibility range.

Go also strictly follows semantic versioning. Thus the implicit exclusive upper bound is the next major version. This assumes that all minor and patch releases are backwards compatibile and not breaking.

Dependency resolution in Go uses minimum version selection. That means the minimum requirements of all dependencies are evaluated and highest minimums are selected. In principle, this minimum version selection should be time invariant since the oldest versions of the compatible dependencies are used

While the minimum versions specified in go.mod are not necessarily the version of the dependencies used, they can be resolved to the versions used irrespective of time or later versions of dependencies being released.

Other languages do not use minimum version selection. Their package resolution often tries to retrieve the latest compatible dependency. Thus a lock file is needed.

Python packages in particular do not follow semantic versioning. Thus ranges are critical in a pyproject.toml.

In summary, the "manifests" files that the original author describes are configuration files. In some languages, or more accurately their package management schemes, they can also be lock files, true manifests, due to version semantics. If those semantics are absent, then lock files are necessary for compatibility.

Re: Go.sum is not a lockfile

#35

The pyproject.toml, package.json, and Cargo.toml are declarative project configuration files. While the Rust community refers to Cargo.toml as a manifest, it is not a comprehensive and detailed list of a build. That is the lock file. While go.mod does not allow for explicit version ranges, the versions given are the minimum versions. In other words, the versions given are the lower bound of the compatibility range. G…

That’s very interesting. Most systems I know would pick the highest versions allowed by the ranges. In maven and gradle, for example, at least by default they choose the highest versions allowed. Even if no version range is used, it picks the highest choice even across major versions, which I always thought was completely broken. What does go do if you have two transitive dependency versions whose allowed major is different?

Re: Go.sum is not a lockfile

#36
post #35

The pyproject.toml, package.json, and Cargo.toml are declarative project configuration files. While the Rust community refers to Cargo.toml as a manifest, it is not a comprehensive and detailed list of a build. That is the lock file. While go.mod does not allow for explicit version ranges, the versions given are the minimum versions. In other words, the versions given are the lower bound of the compatibility range. G…

That’s very interesting. Most systems I know would pick the highest versions allowed by the ranges. In maven and gradle, for example, at least by default they choose the highest versions allowed. Even if no version range is used, it picks the highest choice even across major versions, which I always thought was completely broken. What does go do if you have two transitive dependency versions whose allowed major is di…

In some sense, Go does not allow you to change the major version. Packages with the same name but different major versions are treated as different packages.

Re: Go.sum is not a lockfile

#37
post #11

Earlier quoted context omitted.

> No, it does not. Minimum version selection means that the libraries will at least be that version, but it could be substituted for a later version if a transient dependency asks for such. No? All dependencies - direct and indirect - are listed in your go.mod. Your module - as is - depends on nothing else. And those exact versions will be used to build it, if yours is the main module . If your module is used as a de…

But a “version” as listed in go.mod may have different hashes over time, if tags are changed. That’s the issue.

This doesn't make any sense to me.

If you wanted to verify the contents of a dependency, you would want to check go.sum. That's what it is there for, after all. So if you wanted to fetch the dependencies, then you would want to use it to verify hashes.

If all you care about the is the versions of dependencies, you really can (and should) trust go.mod alone. You can do this because there are multiple overlapping mechanisms that all ensure that a tag is immutable once it is used:

- The Go CLI tools will of course use the go.sum file to validate that a given published version of a module can never change (at least since it was depended on, but it also is complementary with the features below as well, so it can be even better than that.)

- Let's say you rm the go.sum file. That's OK. They also default to using the Go Sum DB to verify that a given published version of a module can never change. So if a module has ever been `go get`'d by a client with the Sum DB enabled and it's publicly accessible, then it should be added to the Sum DB, and future changes to tags will cause it to be rejected.

- And even then, the module proxy is used by default too, so as soon as a published version is used by anyone, it will wind up in the proxy as long as its under a suitable license. Which means that even if you go and overwrite a tag, almost nobody will ever actually see this.

The downside is obviously all of this centralized infrastructure that is depended on, but I think it winds up being the best tradeoff; none of it is a hard dependency, even for the "dependencies should be immutable" aspect thanks to go.sum files. Instead it mostly helps dependency resolution remain fast and reproducible. Most language ecosystems have a hard dependency on centralized infrastructure, whether it is a centralized package manager service like NPM or a centralized repository on GitHub, whereas the centralized infrastructure with Go is strictly complementary and you can even use alternative instances if you want.

But digression aside, because of that, you can trust the version numbers in go.mod.

Re: Go.sum is not a lockfile

#39

The pyproject.toml, package.json, and Cargo.toml are declarative project configuration files. While the Rust community refers to Cargo.toml as a manifest, it is not a comprehensive and detailed list of a build. That is the lock file. While go.mod does not allow for explicit version ranges, the versions given are the minimum versions. In other words, the versions given are the lower bound of the compatibility range. G…

> While the minimum versions specified in go.mod are not necessarily the version of the dependencies used

This has not been true since Go 1.17 with the default -mod=readonly, which is why go.mod is a reliable lockfile.

Re: Go.sum is not a lockfile

#40
post #11

Earlier quoted context omitted.

> No, it does not. Minimum version selection means that the libraries will at least be that version, but it could be substituted for a later version if a transient dependency asks for such. No? All dependencies - direct and indirect - are listed in your go.mod. Your module - as is - depends on nothing else. And those exact versions will be used to build it, if yours is the main module . If your module is used as a de…

But a “version” as listed in go.mod may have different hashes over time, if tags are changed. That’s the issue.

Conversely, I can say that an hash being in go.sum doesn't mean it will be used for anything.

Only that if the corresponding version does get used, and the hash doesn't match, you get an error. But you can have multiple versions of the same dep in your go.sum - or none at all - and this has no bearing on what version gets picked when you build your module.

The version that does get picked is the one in go.mod of the main module, period; go.sum, if it exists, assists hash verification.

Yes, if you want a lockfile in the npm sense, you need both.

But a Go module does not get built with new transitive dependencies (as was claimed) unless they're listed in some go.mod; go.sum is irrelevant for that.

Post reply on HN