Successfully Merging the Work of 1000 Developers
engineering.shopify.com
Successfully Merging the Work of 1000 Developers
1–10 of 108 posts
Re: Successfully Merging the Work of 1000 Developers
#2It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc.
Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actually hear about their work process as this whole article could be summed up as "not running CI on branches before merging their commits to master is a great way to ruin master".
Re: Successfully Merging the Work of 1000 Developers
#3It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…
Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what is "current" will not be when the branch is ready to merge.
Perhaps this problem is what microservices are meant to solve. When you can't coherently integrate code fast enough, attack the bottleneck (master) by splitting it (multiple services).
Re: Successfully Merging the Work of 1000 Developers
#4It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…
Shopify always runs CI on branches before merging to master. Everything this article describes is in addition to that, in order to deal with the problems the article talks about at "merge to master" time, like 2 merged PRs failing or a stale PR that passed on branch but fails on master due changes.
At this scale you need to be deploying constantly, otherwise deploys are hundreds of commits large and its impossible to triage - what PR in the deploy broke something, is it even safe to rollback, etc. That is the primary reason to automate deploys and manage the deploy queue.
Re: Successfully Merging the Work of 1000 Developers
#5It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…
Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…
> Perhaps this problem is what microservices are meant to solve.
Kinda. Microservices have always been an organizational solution; they're a way to shard your company's work output. Usually that's API contracts, but whatever mechanisms are bottlenecked on the work output is affected, including how many concurrent builds are running due to how many people are touching the code at the same time.
Re: Successfully Merging the Work of 1000 Developers
#6It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…
Literally the definition of CI is to run on master or release branches a few times a day, not on every dev branch.
Re: Successfully Merging the Work of 1000 Developers
#7It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…
Hi, Author here!
Pull requests are our unit of work, and the queue was created to support all pull requests. We do have feature flags as a tool, but we let our developers make the judgment call on how their changes should be rolled out.
Re: Successfully Merging the Work of 1000 Developers
#8It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…
We (Shopify) still run the full CI on each development branch as the article mentions:
> We check if Branch CI has passed and if the pull request has been approved by a reviewer before adding the pull request to the queue
Re: Successfully Merging the Work of 1000 Developers
#9It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…
Literally the definition of CI is to run on master or release branches a few times a day, not on every dev branch. https://en.wikipedia.org/wiki/Continuous_integration
The definition is sourced from here[0], and says "Each check-in is then verified by an automated build, allowing teams to detect problems early.". It's not a hard rule to "not run on every dev branch".
"continuous integration" is often just "npm ci && npm run build", and sometimes "npm run test" (or similar for your language). For products that don't make any remote API calls (or when they use a faker service), most of this is done on the same machine and costs very little to do on every commit, making it easier to precisely define which commit broke something.
Re: Successfully Merging the Work of 1000 Developers
#10It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…
Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…
Yet it seems like large companies mostly prefer monorepos, so while it takes investment to have such a monorepo, it seems the benefits are worth the investment.