Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go Upgrade Checklist

hakann.substack.com

Post reply on HN
Print view View original

11–20 of 20 posts

Re: Go Upgrade Checklist

#11

Post by physicles »

nine_k wrote: post #9
physicles wrote: post #4

If you're working at a large company and downtime is extremely expensive, this checklist is a good guide. Otherwise, if you have good test coverage, you can get by with something simpler. It's super rare to have a breaking change in go. We do quarterly upgrades of all services in a monorepo (about 20-30). The steps are basically this: - Upgrade all dependencies to their latest versions, fixing build and test breaks (…

Quarterly upgrades, four years: 16 upgrades. A dozen build breaks means that 75% upgrades face a build break. Since it's over the total number of builds for 20-30 services, it should not be that bad; instead, sometimes there happened a completely uneventful upgrade of everything!

You’re right. Going back over some notes (including go.mod’s history), we’ve been at this for six years, not four. And a dozen build breaks is probably an overestimate — it’s more like one every 12-18 months. Most upgrades are uneventful, everything just continues working.

Re: Go Upgrade Checklist

#12

Post by skunkworker »

jzwinck wrote: post #8
et1337 wrote: post #6

I’ll add an item that is not yet on our checklist but has already bitten us several times: check your code generation. Since code generation is so popular in the Go ecosystem, we’ve got 5 or 6 different codegen tools that update on various timelines. Twice now we’ve gone through a checklist similar to this article, patted ourselves on the back, and a week later found out no one can regenerate any code.

This is one reason why code generation should run as part of the build process, every time. Even if you decide to check-in the generated code for visibility.

The internal tools I've used regenerate the code as part of the CI process, and will fail the pipeline if the regenerated code has dirtied the git tree.

Re: Go Upgrade Checklist

#13

Post by zbentley »

jzwinck wrote: post #8
et1337 wrote: post #6

I’ll add an item that is not yet on our checklist but has already bitten us several times: check your code generation. Since code generation is so popular in the Go ecosystem, we’ve got 5 or 6 different codegen tools that update on various timelines. Twice now we’ve gone through a checklist similar to this article, patted ourselves on the back, and a week later found out no one can regenerate any code.

This is one reason why code generation should run as part of the build process, every time. Even if you decide to check-in the generated code for visibility.

> Even if you decide to check-in the generated code for visibility.

I prefer checking it in by default (generated and checked in by users; CI failing if re-generation during the build generates diff).

It enables much simpler debugging collaboration ("do you have diff in the gen/ dir when you try to repro this bug? I don't."), mistaken-checkin prevention ("did you accidentally run protoc on the wrong version before adding files to this commit? CI's failing because it sees changes in gen/ without changes to requirements or .proto files") and easier verification of upgrades just like this one.

With the ability to use .gitattributes to suppress generated diff visibility by default widely supported (if not well-standardized) across Git repo management platforms, the drawbacks of checking in generated sources are minimal.

Re: Go Upgrade Checklist

#14

Post by peterldowns »

skunkworker wrote: post #12
jzwinck wrote: post #8

Earlier quoted context omitted.

This is one reason why code generation should run as part of the build process, every time. Even if you decide to check-in the generated code for visibility.

The internal tools I've used regenerate the code as part of the CI process, and will fail the pipeline if the regenerated code has dirtied the git tree.

this is the way

Re: Go Upgrade Checklist

#15

Post by tapirl »

> Update go.mod.

Be careful after you did this. Go has changed for-loop semantics since Go 1.22. When you change the go version to 1.22+ from 1.22-, you Go code has a probability to being broken: https://go101.org/blog/2024-03-01-for-loop-semantic-changes-... (It is long. A short important summary is here: https://github.com/golang/go/issues/66156)

Currently (Go 1.24), the official team has not published a tool to identify all of the breaking cases caused by this change. So you might need to check the code by your eyes.

Re: Go Upgrade Checklist

#16

Post by lopkeny12ko »

Another suggestion: if your monorepo's service packaging is sufficiently uniform, build every service against both Go versions, package both binaries into the deploy artifact, and install a feature flag that lets you select which binary to boot when the service starts. This also lets you canary an arbitrary percentage of the fleet with the new Go version, and you can execute a version rollback by redeploying (without needing to revert any commits).

Re: Go Upgrade Checklist

#17

Post by thiht »

tapirl wrote: post #15

> Update go.mod. Be careful after you did this. Go has changed for-loop semantics since Go 1.22. When you change the go version to 1.22+ from 1.22-, you Go code has a probability to being broken: https://go101.org/blog/2024-03-01-for-loop-semantic-changes-... (It is long. A short important summary is here: https://github.com/golang/go/issues/66156 ) Currently (Go 1.24), the official team has not published a tool to i…

I'm not sure you can actually break code with the new for-loop semantics (I mean, in real life situations). It can probably fix some buggy code in the wild, but I have a hard time believing anyone would voluntarily write code relying on the old semantics of loop variables being reassigned instead of reinitialized.

Honestly I think this is a non-issue.

Re: Go Upgrade Checklist

#18

Post by tapirl »

thiht wrote: post #17
tapirl wrote: post #15

> Update go.mod. Be careful after you did this. Go has changed for-loop semantics since Go 1.22. When you change the go version to 1.22+ from 1.22-, you Go code has a probability to being broken: https://go101.org/blog/2024-03-01-for-loop-semantic-changes-... (It is long. A short important summary is here: https://github.com/golang/go/issues/66156 ) Currently (Go 1.24), the official team has not published a tool to i…

I'm not sure you can actually break code with the new for-loop semantics (I mean, in real life situations). It can probably fix some buggy code in the wild, but I have a hard time believing anyone would voluntarily write code relying on the old semantics of loop variables being reassigned instead of reinitialized. Honestly I think this is a non-issue.

> I'm not sure you can actually break code with the new for-loop semantics (I mean, in real life situations).

The linked issue thread provides real life cases.

> It can probably fix some buggy code in the wild, but I have a hard time believing anyone would voluntarily write code relying on the old semantics of loop variables being reassigned instead of reinitialized.

For "for-range" loops, you might be correct. However, for 3-clause-for loops, your opinions lack substantial evidence to support them.

> Honestly I think this is a non-issue.

You can think anything. But the facts are there. There are at least 3 important facts of the semantic change on 3-clause-for loops:

1. The new semantics of 3-clause-for loops are more error prone for concurrency programming.

2. The new semantics of 3-clause-for loops may silently downgrade Go code performance.

3. The expected benefits of the change are actually so minuscule that they can be disregarded. On the other hand, the drawbacks of the change are huge.

Re: Go Upgrade Checklist

#19

Post by hakann »

crockeo wrote: post #3
peterldowns wrote: post #2

Out of curiosity, were you dealing with microservices defined within a monorepo, or microservices each in their own repo? The steps here: > Build your binaries with the new version. Go through the build errors if any. > Run all the unit tests with the new version. Go through the test failures. are a lot easier in a monorepo. Separately, I've experienced frequent breaking changes in the golangci-lint configuration fil…

Not Hakan, but I was working closely with him at the time. Lyft was on a microservice many-repo setup, and we did pin the version of golangci-lint. I've found it's actually not so bad to do this kind of work across many repos, as long as you have the tooling to apply the same change to any number of codebases all at once. Our strategy was typically: - Write an idempotent codemod to do an upgrade. This is easy as long…

Thanks crockeo! You are absolutely correct!

Re: Go Upgrade Checklist

#20

Post by peterldowns »

mseepgood wrote: post #7
peterldowns wrote: post #2

Out of curiosity, were you dealing with microservices defined within a monorepo, or microservices each in their own repo? The steps here: > Build your binaries with the new version. Go through the build errors if any. > Run all the unit tests with the new version. Go through the test failures. are a lot easier in a monorepo. Separately, I've experienced frequent breaking changes in the golangci-lint configuration fil…

> Bumping go.mod and downloading the new version of go is usually all it takes! Doesn't it auto-download when you bump go.mod nowadays?

It might — I use nix to pin my development dependencies, including my `go` binary, so I'm not sure.
Post reply on HN