Live data from Hacker News

Go.sum is not a lockfile

words.filippo.io

51–60 of 81 posts

Re: Go.sum is not a lockfile

#51

Earlier quoted context omitted.

These libraries are honestly so bad that whenever I have to interact with them I just split those dependencies into a separate binary :)

I dabbled in Kubernetes at work a few years ago. I was shocked at the state of their go.mod files. Especially because I had heard Kubernetes used as an example of great Go software. Right off the bat I needed to copy a dozen or so `replace` statements into my go.mod. I was sure I just didn’t understand something and spent an hour or so looking for the “right way” only to discover other open source controllers that ha…

Definitely do not look at Kubernetes as a good example of go code and especially not how it handles deps (btw, it only relatively recently switched to go mods).

Not to say it is a bad project. Not at all.

Re: Go.sum is not a lockfile

#54
Two very similar things are presented as though they are different (go.mod and lockfiles, not go.sum) for the purpose of sneering at one of them, when both are essentially the same. 'Ignored by downstream dependents' is not any less true of go.mod than of lockfiles. In both cases a later version can be demanded, overriding the earlier version, potentially breaking your code.

Re: Go.sum is not a lockfile

#55

Earlier quoted context omitted.

These libraries are honestly so bad that whenever I have to interact with them I just split those dependencies into a separate binary :)

I dabbled in Kubernetes at work a few years ago. I was shocked at the state of their go.mod files. Especially because I had heard Kubernetes used as an example of great Go software. Right off the bat I needed to copy a dozen or so `replace` statements into my go.mod. I was sure I just didn’t understand something and spent an hour or so looking for the “right way” only to discover other open source controllers that ha…

I think a lot of people assume Kubernetes must be a good Go example, because it's so big and successful. But it started life as a Java project and was ported into Go, and that shows in some of the architecture, Also, in general across all languages, picking the absolute largest projects you can is often not a great idea in terms of copying design unless you too are going to be that large, e.g., I wouldn't suggest using Firefox as an example of C++ necessarily. Such projects always end up developing solutions to problems you will never have, and solving problems we don't have is one of the most common mistakes software developers make.

Re: Go.sum is not a lockfile

#56
post #48
post #46

Earlier quoted context omitted.

> The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module Incorrect. Checksums are stored forever, in a Merkle Tree, meaning if the proxy were to ever delete a checksum, it would be detected (and yes, people like me are checking - https://sourcespotter.com/sumdb ). Like any code host, the proxy does not guarantee that the code for a module will be available fore…

Ah, my mistake. I had read in the FAQ that it does not guarantee that data is stored forever, but overlooked the part about preserving checksums specifically.

To be very pedantic, there are two separate services: The module proxy (proxy.golang.org) serves cached modules and makes no guarantees about how long cache entries are kept. The sum database (sum.golang.org) serves module checksums, which are kept forever in a Merkle tree/transparency log.

Re: Go.sum is not a lockfile

#57

Earlier quoted context omitted.

These libraries are honestly so bad that whenever I have to interact with them I just split those dependencies into a separate binary :)

I dabbled in Kubernetes at work a few years ago. I was shocked at the state of their go.mod files. Especially because I had heard Kubernetes used as an example of great Go software. Right off the bat I needed to copy a dozen or so `replace` statements into my go.mod. I was sure I just didn’t understand something and spent an hour or so looking for the “right way” only to discover other open source controllers that ha…

They write it in space shuttle mode. Which means that it’s all got too many components from too many makers all kludged together and then one piece falls off and the whole thing blows up in production.

Re: Go.sum is not a lockfile

#58

Earlier quoted context omitted.

These libraries are honestly so bad that whenever I have to interact with them I just split those dependencies into a separate binary :)

I just write my own, small, focused client for these AWS services since the SDKs are generally just so unergonomic. With vibe coding becoming a thing it's become even easier to do that.

You may already know these things, but for others who may not: the AWS SDK for Go suffers from shockingly bad discoverability.

For example, take the S3 library, github.com/aws/aws-sdk-go-v2/service/s3. If you have an s3.Client and you look at e.g. the ListObjectsV2 method, you might have no idea that there is a ListObjectsV2Paginator which makes it much easier to use, because nowhere in the method docs is it mentioned. Indeed, most operations that paginate have more ergonomic paginators, but none of them tell you this.

But that isn't even the worst of it. Say you want to download or upload a file to S3. If you haven't worked with AWS for other languages, you might think that you just do GetObject and PutObject. And yes, for small files, that's generally fine. But for large files you want to use resumable downloads and multipart uploads. So you look and lo, there is no simple way to do this in the AWS SDK for Go. But actually, there is! It's in a totally unrelated and unlinked package, called github.com/aws/aws-sdk-go-v2/feature/s3/manager.

Now you're getting some religion, so you ask "what are the other so-called 'feature' packages?" and you try to browse pkg.go.dev at the github.com/aws/aws-sdk-go-v2/feature level but nope, that's not a Go module so there's nothing to see there. In fact, you can't even browse the other s3 features, never mind find out what other services have features. Fortunately, you can browse their GitHub repo at least: https://github.com/aws/aws-sdk-go-v2/tree/main/feature

It's quite clear that they use poorly thought-out cross-language codegen for this, which partly explains the lack of ergonomics, but also shows that they don't much care whether you use their stuff properly.

Re: Go.sum is not a lockfile

#59
post #46
post #43

Earlier quoted context omitted.

> its stored forever in the proxy cache This is mistaken. The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module. From the outside, we would expect that their policy is to only ever delete checksums for modules that haven't been fetched in a long time. But in general, you should not base your security model on the notion that these checksums are stored permanen…

> The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module Incorrect. Checksums are stored forever, in a Merkle Tree, meaning if the proxy were to ever delete a checksum, it would be detected (and yes, people like me are checking - https://sourcespotter.com/sumdb ). Like any code host, the proxy does not guarantee that the code for a module will be available fore…

Ok. So to answer the question whether the code for v1.0.0 that I downloaded today is the same as I downloaded yesterday (or whether the code that I get is the same as the one my coworker is getting) you basically have to trust Google.

Re: Go.sum is not a lockfile

#60

Earlier quoted context omitted.

Let's assume I publish a github repo with some go code, and tag a particular commit with tag v1.0.0. People start using it and put v1.0.0 into their go.mod file. They use the golang proxy to fetch the code (and that proxy does the "verification", according to your comment). Now I delete the v1.0.0 tag and re-create the tag to point to different (malicious) commit. Will the golang proxy notice? How does it verify that…

yes. From my understanding its stored forever in the proxy cache and your new tag will never be fetched by users who go through the language's centralized infrastructure (i.e. proxy). go can also validate the checksums (go.sum) against the languages central infrastructure that associates version->checksums. i.e. if you cut a release, realize you made a mistake and try to fix it quitely, no user will ever see it if ev…

This makes the assumption that everyone uses the default proxy, which is not the case
Post reply on HN