Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Successfully Merging the Work of 1000 Developers

engineering.shopify.com

Post reply on HN
Print view View original

71–80 of 108 posts

Re: Successfully Merging the Work of 1000 Developers

#71

Post by robocat »

peterwwillis wrote: post #65
dkoston wrote: post #2

It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…

> Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what is "current" will not be when the branch is ready…

Google don't solve it your way: https://news.ycombinator.com/item?id=21586180

Re: Successfully Merging the Work of 1000 Developers

#72

Post by metajack »

omeze wrote: post #43
DilumAluthge wrote: post #28

This is exactly the kind of workflow that Bors ( https://github.com/bors-ng/bors-ng ) automates. In addition to Bors, there are a number of apps and services that automate this kind of workflow. Here is an incomplete list: https://forum.bors.tech/t/other-apps-that-implement-the-nrsr... Edited to add: Graydon Hoare (creator of Rust) called this the Not Rocket Science Rule Of Software Engineering (NRSROSE): "automatica…

Bors when introduced at my workplace was summarily disabled after 2 separate incidents where it was improperly applying commit messages (which we patched internally) and improperly applying patches (at which point we disabled it and explored other solutions). Combined with the merge latency overhead, I would be extremely hesitant to advocate running it at scale.

At this point there are like 4 implementations of Graydon's bors. Which one did you use?

The Servo and Rust projects have used a bors* for many years and have had few problems.

* Servo uses the https://github.com/servo/homu fork of the Homu rewrite of Graydon's original code, and Rust uses a slightly different fork at https://github.com/rust-lang/homu.

Re: Successfully Merging the Work of 1000 Developers

#73

Post by metajack »

jacktli wrote: post #40
DilumAluthge wrote: post #28

This is exactly the kind of workflow that Bors ( https://github.com/bors-ng/bors-ng ) automates. In addition to Bors, there are a number of apps and services that automate this kind of workflow. Here is an incomplete list: https://forum.bors.tech/t/other-apps-that-implement-the-nrsr... Edited to add: Graydon Hoare (creator of Rust) called this the Not Rocket Science Rule Of Software Engineering (NRSROSE): "automatica…

The limitation of Bors for us was throughput, we were more interested in testing multiple simultaneous pull requests merging at the same time, rather than testing against the latest master. With our CI times and PR volume, by the time any CI run completes, master would have drastically changed.

Sequentializing landing so that you can ensure passing tests is the main feature of bors. The normal GitHub CI flow is already what you wanted apparently.

In a bors workflow, bors is the only thing allowed to push to master, so master cannot get out of date.

The Rust projects solves throughput with rollups, which are semi-automated. It would be nice if someone would write fully automated rollup support into a bors, but alas, no one has tried that I know of.

Re: Successfully Merging the Work of 1000 Developers

#74

Post by DougBTX »

demosito666 wrote: post #66
DilumAluthge wrote: post #28

This is exactly the kind of workflow that Bors ( https://github.com/bors-ng/bors-ng ) automates. In addition to Bors, there are a number of apps and services that automate this kind of workflow. Here is an incomplete list: https://forum.bors.tech/t/other-apps-that-implement-the-nrsr... Edited to add: Graydon Hoare (creator of Rust) called this the Not Rocket Science Rule Of Software Engineering (NRSROSE): "automatica…

I've read the description, but I fail to understand how this is different from just merging current master into the PR and running integration on that before merging in back into master? This can be done with 10 lines of groovy in Jenkins. Also the exact workflow described with staging branch and batch merge is probably another 20-30 lines.

That process assumes nothing gets merged during the integration test run. The script could check to see if the target branch has changed in the mean time, and fail the merge if it has, then I think it would be the same.

Re: Successfully Merging the Work of 1000 Developers

#75

Post by metajack »

chdsbd wrote: post #15

A bit of shameless self promotion: I built a more basic merge bot for GitHub that efficiently updates and merges PRs because we were wasting a ton of time keeping branches updated at work. https://github.com/chdsbd/kodiak

Thanks for the chart comparing it to many alternatives. It both makes it clear what new things Kodiak does and what alternatives one could look at.

Re: Successfully Merging the Work of 1000 Developers

#76

Post by Eiriksmal »

chdsbd wrote: post #15

A bit of shameless self promotion: I built a more basic merge bot for GitHub that efficiently updates and merges PRs because we were wasting a ton of time keeping branches updated at work. https://github.com/chdsbd/kodiak

We started using Kodiak after the Auto Rebase bot was discontinued a few weeks ago. Other than confusing me with a normal merge when I thought it had been configured to do squash-merges, it works great. Thanks for releasing it!

Re: Successfully Merging the Work of 1000 Developers

#77

Post by dkoston »

wvanbergen wrote: post #70
dkoston wrote: post #58

Earlier quoted context omitted.

Thanks, I was hoping for more of this in the blog post. Since tools are just an expression of process/policy, it’s more interesting to here about the process and why than it is about building “yet another CD tool”. Appreciate the thoughtful and thorough response. The major pain point I agree with on develop is changing the defaults to merge to that rather than master. It’s a shame this is not easier to do in git/gith…

> I guess I’m not fully understanding how a queue prevents this. Since you don’t have a full picture of the state of master until something is merged from the queue, how do the CI checks in the queue prevent things that branch-based CI checks wouldn’t prevent in a “develop” branch? With branches and develop, pull requests remain open until they can be assured they merge properly with develop as well. The trick of the…

I think I see where you are coming from. Being as we use different tools, we wouldn’t allow a pull to be merged if it wasn’t up-to-date with master which is similar but a different approach. You’ll have to check at merge time because getting up-to-date could take a while and master could have changed. Jenkins does this and it can be done in other CI/CD systems with a bit of custom code.

I’d imagine at 1,000 developers and with a monolithic codebase, you’re looking to minimize test runs both from a time and cost of runners perspective.

You may also want to look into Zuul or Bazel if cost of test suite runs is a factor in coming to this solution.

Re: Successfully Merging the Work of 1000 Developers

#78

Post by peterwwillis »

robocat wrote: post #71
peterwwillis wrote: post #65

Earlier quoted context omitted.

> Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what is "current" will not be when the branch is ready…

Google don't solve it your way: https://news.ycombinator.com/item?id=21586180

Yes, that's correct, Google invented its own proprietary distributed object store and distributed version control system and distributed Linux-only filesystem and distributed build-and-test-system to work with a single SDLC that its entire company must follow strictly to release anything, just so it could keep using a single repository.

What's your point?

Re: Successfully Merging the Work of 1000 Developers

#79

Post by wvanbergen »

dkoston wrote: post #77
wvanbergen wrote: post #70

Earlier quoted context omitted.

> I guess I’m not fully understanding how a queue prevents this. Since you don’t have a full picture of the state of master until something is merged from the queue, how do the CI checks in the queue prevent things that branch-based CI checks wouldn’t prevent in a “develop” branch? With branches and develop, pull requests remain open until they can be assured they merge properly with develop as well. The trick of the…

I think I see where you are coming from. Being as we use different tools, we wouldn’t allow a pull to be merged if it wasn’t up-to-date with master which is similar but a different approach. You’ll have to check at merge time because getting up-to-date could take a while and master could have changed. Jenkins does this and it can be done in other CI/CD systems with a bit of custom code. I’d imagine at 1,000 developer…

> Being as we use different tools, we wouldn’t allow a pull to be merged if it wasn’t up-to-date with master which is similar but a different approach

That wouldn't work for us due to the amount of changes we need to ship. If you rebase your branch and wait for CI to come back green, chances are another PR will have merged in the mean time, which means your rebased branch is no longer up to date with master. You end up stuck in a rebase cycle.

For this reason, we have no choice but to batch PRs, which is what the merge queue tool does. Faster CI will reduce this problem and we're working on that as well, but won't fully solve this.

Re: Successfully Merging the Work of 1000 Developers

#80

Post by peterwwillis »

tudelo wrote: post #10
latortuga wrote: post #3

Earlier quoted context omitted.

Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…

Yet it seems like large companies mostly prefer monorepos, so while it takes investment to have such a monorepo, it seems the benefits are worth the investment.

This is appeal to accomplishment fallacy. Because large companies have a lot of money, whatever they do must be great. But this is false - they do what they do because they are large companies, not because it is a good idea.

At scale, managing complexity can require either a lot of coordination, or a lot of careful planning. Large companies (especially tech companies) don't do either well, so they pick architectures that remove choices, and iterate on them until they are workable. And they have the money and workforce to do it.

Post reply on HN