Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Successfully Merging the Work of 1000 Developers

engineering.shopify.com

Post reply on HN
Print view View original

21–30 of 108 posts

Re: Successfully Merging the Work of 1000 Developers

#21

Post by w-m »

jrockway wrote: post #13
latortuga wrote: post #3

Earlier quoted context omitted.

Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…

We didn't have a merge queue at Google. You rebased if there was a merge conflict, ran through CI again, and hoped there wasn't another merge conflict. I think I ran into merge conflicts maybe once a year, if that. I think the success of this system breaks down into several parts: 1) Yup, microservices. You could submit your proto change, which would affect all clients, before actually implementing the code that used…

Point 2 is very important and very hard to get right. For unit tests, there is a clear dependency on the code and you can easily just run a subset of the tests. But wouldn't you have to run any system and integration tests of the affected module, as it's not clear what effects the code change can have? This will blow up CI times again. How did Google deal with this?

Re: Successfully Merging the Work of 1000 Developers

#22

Post by dkoston »

tekstar wrote: post #4
dkoston wrote: post #2

It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…

Shopify always runs CI on branches before merging to master. Everything this article describes is in addition to that, in order to deal with the problems the article talks about at "merge to master" time, like 2 merged PRs failing or a stale PR that passed on branch but fails on master due changes. At this scale you need to be deploying constantly, otherwise deploys are hundreds of commits large and its impossible to…

Sorry if my comment was unclear. I consider the queue to be a “branch” as well. Many people use a “develop” branch instead of a queue in this instance. The queue appears designed to allow arbitrary selection rather than merging in order (though the new solution with CD seems generally in order)

Totally agree that CD is required with this many commits. It’s commonplace on teams with many fewer developers. Was surprised to see you folks roll your own workflows rather than using other systems.

Would also be interesting to see if you tag commits that go to master in instrumentation systems so you have visibility into production metrics and can correlate them with what code was running at the time.

Re: Successfully Merging the Work of 1000 Developers

#23

Post by dkoston »

rb808 wrote: post #6
dkoston wrote: post #2

It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…

Literally the definition of CI is to run on master or release branches a few times a day, not on every dev branch. https://en.wikipedia.org/wiki/Continuous_integration

Rather than being pedantic about the definition, maybe you could share your experiences with why that’s superior to validating each branch? Being dogmatic about a definition rather than experimenting with works best in production at your business seems illogical.

Re: Successfully Merging the Work of 1000 Developers

#25

Post by jrockway »

sshumaker wrote: post #18
jrockway wrote: post #13

Earlier quoted context omitted.

We didn't have a merge queue at Google. You rebased if there was a merge conflict, ran through CI again, and hoped there wasn't another merge conflict. I think I ran into merge conflicts maybe once a year, if that. I think the success of this system breaks down into several parts: 1) Yup, microservices. You could submit your proto change, which would affect all clients, before actually implementing the code that used…

Xoogler here. When I left in 2015 there were definitely teams that used merge queues (i.e. TAP presubmit). Generally these were teams with a more monolithic architecture, like YouTube that had a massive Python mono.

I guess TAP presubmit might be a merge queue... but it seems different from this. There was no requirement that some mechanical system checked that tests passed before your merged your CL. You could merge any code whenever it was approved. If you felt like running the tests, good for you. TAP presubmit is just that mechanical system that runs your tests before executing the merge. That seems like traditional CI to me, not a merge queue.

Jenkins with a Github plugin behaves almost exactly like this system. Every PR basically has tests run 3 times; once for the branch that the PR is on, once for your branch merged to master, and then once after you do the merge and submit it. TAP presubmit did the "once for your branch" and TAP did the post-merge CI.

TAP presubmit didn't really check that the resulting merge was sound, so you would see TAP presubmit pass, your change get merged, and then have the build break anyway because of the race condition. A merge queue would not have this race condition... so I'm not sure Shopify has one either. The more I think about it the more it sounds like they just rewrote Jenkins. (And for that, I don't blame them.)

Re: Successfully Merging the Work of 1000 Developers

#26

Post by eejjjj82 »

This is how google manages these changes:

https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...

this was written 3 years ago and scale has only grown. But at the time the numbers quoted in the article are 25,000 authors / day w/ 16,000 commits and another 24,000 commits / day from automated sources

Re: Successfully Merging the Work of 1000 Developers

#27

Post by dkoston »

jacktli wrote: post #7
dkoston wrote: post #2

It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…

Hi, Author here! Pull requests are our unit of work, and the queue was created to support all pull requests. We do have feature flags as a tool, but we let our developers make the judgment call on how their changes should be rolled out.

Interesting. It seems like you have a very flexible process of how to launch code which could contribute to issues with visibility and rollbacks.

I’m curious as to why you had a queue instead of a develop branch before moving to CD? Was this to allow arbitrary commits to be launched to production rather than getting them batched by time?

Re: Successfully Merging the Work of 1000 Developers

#28

Post by DilumAluthge »

This is exactly the kind of workflow that Bors (https://github.com/bors-ng/bors-ng) automates.

In addition to Bors, there are a number of apps and services that automate this kind of workflow. Here is an incomplete list: https://forum.bors.tech/t/other-apps-that-implement-the-nrsr...

Edited to add: Graydon Hoare (creator of Rust) called this the Not Rocket Science Rule Of Software Engineering (NRSROSE): "automatically maintain a repository of code that always passes all the tests" - https://graydon2.dreamwidth.org/1597.html

Disclaimer: I have contributed code to Bors.

Re: Successfully Merging the Work of 1000 Developers

#29

Post by gav »

adrianN wrote: post #12
latortuga wrote: post #3

Earlier quoted context omitted.

Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…

Microservices don't really help with this. They just force you to think about your interfaces, but you should do that in a monolith too. If you interfaces are reasonably stable, merging is unlikely to break master if the branch was green before, if your interfaces change rapidly you get problems with microservices too, just one level higher up, where you try to integrate them into a usable product.

One of the things I think microservices does help with is thinking about systems being composed of components that are being developed at different velocities and different tolerances of risk.

Imagine an e-commerce site broken into a bunch of services including search and checkout. The search team is making updates daily, trying to improve ranking and drive conversion. The checkout team (assuming that the site is mature and has hit some design equilibrium) may only be releasing changes every couple of months, and if a bug is introduced, the financial impact is a lot higher.

By not bundling the outputs of very different teams together, you can help those that want to "move fast and break things" with their "moving fast" goal, and de-risk breaking everything by reducing the surface area of changes. Microservices-based architectures are a way to reduce friction caused by the structure of your organization and is one outcome of an Inverse Conway Maneuver.

Post reply on HN