Live data from Hacker News

Successfully Merging the Work of 1000 Developers

engineering.shopify.com

11–20 of 108 posts

Re: Successfully Merging the Work of 1000 Developers

#11
post #2

It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…

Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…

This paper might be of interest to you on this very subject:

https://eng.uber.com/research/keeping-master-green-at-scale/

Re: Successfully Merging the Work of 1000 Developers

#12
post #2

It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…

Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…

Microservices don't really help with this. They just force you to think about your interfaces, but you should do that in a monolith too. If you interfaces are reasonably stable, merging is unlikely to break master if the branch was green before, if your interfaces change rapidly you get problems with microservices too, just one level higher up, where you try to integrate them into a usable product.

Re: Successfully Merging the Work of 1000 Developers

#13
post #2

It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…

Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…

We didn't have a merge queue at Google. You rebased if there was a merge conflict, ran through CI again, and hoped there wasn't another merge conflict. I think I ran into merge conflicts maybe once a year, if that.

I think the success of this system breaks down into several parts:

1) Yup, microservices. You could submit your proto change, which would affect all clients, before actually implementing the code that used the new feature. (Or after, in the case of renaming some field from foo to deprecated_foo and refactoring the clients to stop using that field.) That means you could wrangle that change without having to worry about it affecting your actual feature. (Typically proto changes did not cause any breakages since people were very conservative about what changes they would make. Nobody renames all the fields, invalidating dependent code, or renumbers the fields, invalidating all existing messages. You COULD do those things, but nobody ever did.)

2) Clear dependencies in the build system. The CI system only had to run a small set of tests for most changes, because it knew exactly what tests the change would affect. You had to go way out of your way to depend on code without informing the build system. This is very different from every CI system that I've seen outside of Google, which seem to default to running everything and hoping your programming language or build system magically tracks dependencies. It doesn't; Docker for example will happily use random images that it thinks haven't changed, without actually checking if it has changed. (Consider building your app on top of golang:latest. Go is updated, and docker may or may not pull that new base image. Meanwhile, docker will happily clear its build cache if you edit README.md and no code. The result is that 50% of the time you waste 10 minutes rebuilding stuff that didn't change, and 50% of the time you get an outdated build. And nobody seems to care at all!)

3) Being careful about keeping changes small. I don't know what the average CL size is, but I would aim for 100 lines changed rather than 1000 lines changed. This is something that surprised me post-Google, people go away and work for a week and you have a 2000 line PR to review. These are tough to merge and were relatively rare in my experience at Google. It is not always possible to make every change small, but that should be the norm. Figure out how much work you can do in a day, and try to make a CL/PR that is that size. A lot can churn in a week. A lot less churns in a day. If you respected steps 1 and 2, that means your tests will run fast and it's unlikely that your merge will fail between CI and actually merging. If you have 2000 lines of code across 8 services... you'll probably never get it merged. But I am sure that I have successfully merged ginormous changes before, it's just more work.

All in all, my takeaway from this article is that Shopify is huge but I'm surprised that specialized merge tooling was necessary. I wonder what the underlying problem is; do they really have a 1000 developer monolith? Do they not use a proper build system like Bazel?

Re: Successfully Merging the Work of 1000 Developers

#14
post #2

It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…

Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…

> Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with.

Google does it with 50 times the developer count.

> At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build).

True, it is impossible to catch all errors like this, but you can catch almost every error by building and testing it against current master and then merge it with the master 20 minutes later when the build is done. I have seen maybe one build breakage a year being introduced due to this in projects I've worked on, so it isn't a big deal.

Re: Successfully Merging the Work of 1000 Developers

#16
If I read this article right, if anyone breaks any part of the build it breaks for everyone? Doesn't sound very scalable. Shouldn't the main goal be to break up your continuous integration steps so that a person in one end of the company can continue working even if a person in the other end broke his build?

That way you can also add tags for flaky tests etc, to make your builds more reliable.

Edit: I didn't understand before what people had against monorepos, but I guess if you tie an continuous integration builds to each repo then having a monorepo becomes a huge pain point. Are there any open source tools to fix this?

Re: Successfully Merging the Work of 1000 Developers

#17

If I read this article right, if anyone breaks any part of the build it breaks for everyone? Doesn't sound very scalable. Shouldn't the main goal be to break up your continuous integration steps so that a person in one end of the company can continue working even if a person in the other end broke his build? That way you can also add tags for flaky tests etc, to make your builds more reliable. Edit: I didn't understa…

Presumably when a merge breaks the main branch, the merge gets reverted and it is the task of whoever pushed that to fix it. In the mean time the CI server can continue with the next branch/commit in the queue.

Re: Successfully Merging the Work of 1000 Developers

#18

Earlier quoted context omitted.

Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…

We didn't have a merge queue at Google. You rebased if there was a merge conflict, ran through CI again, and hoped there wasn't another merge conflict. I think I ran into merge conflicts maybe once a year, if that. I think the success of this system breaks down into several parts: 1) Yup, microservices. You could submit your proto change, which would affect all clients, before actually implementing the code that used…

Xoogler here. When I left in 2015 there were definitely teams that used merge queues (i.e. TAP presubmit). Generally these were teams with a more monolithic architecture, like YouTube that had a massive Python mono.

Re: Successfully Merging the Work of 1000 Developers

#19

Earlier quoted context omitted.

Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…

> Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. Google does it with 50 times the developer count. > At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). True, it is impossible to catch all errors like this, but you can catch a…

> building and testing it against current master and then merge it with the master 20 minutes later when the build is done.

And I'm pretty sure that is the way Google does it too. Test a commit against current master, if tests are green commit. Then run tests against master again (and I think this stage might not run for every single commit) to see if anything broke on the rare times there was an actual conflict. If that run was red, which should be rare, then you can have the system do a bisect to find the offending commit, or just run all the ones that haven't been individually tested.

Post reply on HN