Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Successfully Merging the Work of 1000 Developers

engineering.shopify.com

Post reply on HN
Print view View original

81–90 of 108 posts

Re: Successfully Merging the Work of 1000 Developers

#81

Post by allthetime »

coldcode wrote: post #30

I find it difficult to imagine why you would need 1000 developers in most single projects. The cost of managing so many people and work streams would seem to overwhelm whatever volume you could get out of them

Yeah, honestly, as a person who has had the ill fortune of having to work with shopify, I find this kind of boasting kind of funny. Every admin console page load takes ten seconds and their backend can't handle a product with more than 100 variants. They clearly got too big too fast and just piled code on code on code until they ended up with what they have today. They should focus on upgrading the quality of their system, take 20 of those developers and start on a complete rebuild, and spend a little less time wrangling 1000 devs.

Re: Successfully Merging the Work of 1000 Developers

#82

Post by [deleted] »

peterwwillis wrote: post #65
dkoston wrote: post #2

It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…

> Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what is "current" will not be when the branch is ready…

[deleted]

Re: Successfully Merging the Work of 1000 Developers

#83

Post by randomidiot666 »

username90 wrote: post #14
latortuga wrote: post #3

Earlier quoted context omitted.

Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…

> Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. Google does it with 50 times the developer count. > At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). True, it is impossible to catch all errors like this, but you can catch a…

You have no idea how Google solved it. Basically everyone with a Monorepo (except Google) implements it as a cargo cult best practice. Mindlessly copying Google without understanding how Google actually does it.

Re: Successfully Merging the Work of 1000 Developers

#84

Post by randomidiot666 »

robocat wrote: post #71
peterwwillis wrote: post #65

Earlier quoted context omitted.

> Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what is "current" will not be when the branch is ready…

Google don't solve it your way: https://news.ycombinator.com/item?id=21586180

You have no idea how Google solved it. Basically everyone with a Monorepo (except Google) implements it as a cargo cult best practice. Mindlessly copying Google without understanding how Google actually does it.

Re: Successfully Merging the Work of 1000 Developers

#85

Post by bob1029 »

latortuga wrote: post #3
dkoston wrote: post #2

It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…

Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…

By virtue of having a queue of PRs that need to test & merge, you could pipeline this thing out pretty substantially.

The implication here being that a queue must be processed in-order, so you will ultimately have a perfect sequence of future commits to speculate against, and can incrementally build up each hypothetical future master state for a test build on one of any number of parallel build agents. As the queue depth grows, you would see higher and higher throughput.

Re: Successfully Merging the Work of 1000 Developers

#87

Post by Ericson2314 »

metajack wrote: post #73
jacktli wrote: post #40

Earlier quoted context omitted.

The limitation of Bors for us was throughput, we were more interested in testing multiple simultaneous pull requests merging at the same time, rather than testing against the latest master. With our CI times and PR volume, by the time any CI run completes, master would have drastically changed.

Sequentializing landing so that you can ensure passing tests is the main feature of bors. The normal GitHub CI flow is already what you wanted apparently. In a bors workflow, bors is the only thing allowed to push to master, so master cannot get out of date. The Rust projects solves throughput with rollups, which are semi-automated. It would be nice if someone would write fully automated rollup support into a bors, b…

You don't need absolute sequentiality for bors's guarantees. You can speculatively build and try to merge multiple PRs in parallel even though only one will "win". That's fine and not thundering herd stupidity of your build system is incremental so you can share work.

None of this is new at all, btw, I'm just regurgitating MVCC from postgresql.

Re: Successfully Merging the Work of 1000 Developers

#88

Post by anon73044 »

tudelo wrote: post #10
latortuga wrote: post #3

Earlier quoted context omitted.

Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…

Yet it seems like large companies mostly prefer monorepos, so while it takes investment to have such a monorepo, it seems the benefits are worth the investment.

Google, Microsoft, Facebook and Twitter prefer monorepos but this is not indicative of most large orgs.

You'll notice that those listed have had to customize or creat new vcs's to meet their needs.

https://news.ycombinator.com/item?id=17605371 https://news.ycombinator.com/item?id=11789182 https://medium.com/@maoberlehner/monorepos-in-the-wild-33c6e...

Re: Successfully Merging the Work of 1000 Developers

#90

Post by dkoston »

wvanbergen wrote: post #79
dkoston wrote: post #77

Earlier quoted context omitted.

I think I see where you are coming from. Being as we use different tools, we wouldn’t allow a pull to be merged if it wasn’t up-to-date with master which is similar but a different approach. You’ll have to check at merge time because getting up-to-date could take a while and master could have changed. Jenkins does this and it can be done in other CI/CD systems with a bit of custom code. I’d imagine at 1,000 developer…

> Being as we use different tools, we wouldn’t allow a pull to be merged if it wasn’t up-to-date with master which is similar but a different approach That wouldn't work for us due to the amount of changes we need to ship. If you rebase your branch and wait for CI to come back green, chances are another PR will have merged in the mean time, which means your rebased branch is no longer up to date with master. You end…

That’s understandable. I’d imagine at some point you’ll need to decouple the monolith a bit in order to work effectively as you scale. Best of luck with the challenge.
Post reply on HN