Successfully Merging the Work of 1000 Developers • Hacker News Board

Re: Successfully Merging the Work of 1000 Developers

#101

Post by pojzon » Thu, Nov 21, 2019, 6:15 AM UTC

byroot wrote: post #57

pojzon wrote: post #42

Earlier quoted context omitted.

That pretty much forces tooling to try to rebase and run CI either way ? If it cannot rebase its up to the developer to fix his branch. Master should be always in a state of release at any moment. I just cannot imagine it was an unknown practice for some.

Except that with the amount of activity on the repo it's simply impossible. Every new merge on master would require to rebase several hundred branches being worked on or awaiting reviewed. Multiply this with the hundreds of commits merged on master every day and you end up with way too much CI jobs to run.

Its not impossible.. thats what they ended up with either way.. And how do I know its possible ? Well I have over 2000 developers working on the same codebase in my current company. I work as a devops and this conversation is stupid..

Re: Successfully Merging the Work of 1000 Developers

#102

Post by [deleted] » Thu, Nov 21, 2019, 7:00 AM UTC

randomidiot666 wrote: post #83

username90 wrote: post #14

Earlier quoted context omitted.

> Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. Google does it with 50 times the developer count. > At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). True, it is impossible to catch all errors like this, but you can catch a…

You have no idea how Google solved it. Basically everyone with a Monorepo (except Google) implements it as a cargo cult best practice. Mindlessly copying Google without understanding how Google actually does it.

[deleted]

Re: Successfully Merging the Work of 1000 Developers

#103

Post by randomidiot666 » Thu, Nov 21, 2019, 7:08 AM UTC

randomidiot666 wrote: post #83

username90 wrote: post #14

Earlier quoted context omitted.

> Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. Google does it with 50 times the developer count. > At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). True, it is impossible to catch all errors like this, but you can catch a…

You have no idea how Google solved it. Basically everyone with a Monorepo (except Google) implements it as a cargo cult best practice. Mindlessly copying Google without understanding how Google actually does it.

To the clowns downvoting my comment: to even suggest that Google does a full automated regression test on the entire codebase on every merge is beyond stupidity. Having a single Git repository with full regression test on every merge is a primitive naive interpretation of how actually Google does it. Their solution is very different and very custom built.

Re: Successfully Merging the Work of 1000 Developers

#104

Post by winrid » Thu, Nov 21, 2019, 7:51 AM UTC

baroffoos wrote: post #98

jacktli wrote: post #96

Earlier quoted context omitted.

It’s about providing a platform for enabling global commerce. Our engineering blog has other posts that goes into technical depth about a lot of the challenges we face, and our solutions. For context, here is some info about Black Friday/Cyber Monday for us last year: https://engineering.shopify.com/blogs/engineering/preparing-...

I understand its a hugely popular service and there is a lot that goes in to something so big but there are bigger websites that run on much much smaller teams of developers. I just don't understand how there is even enough surface area on the app that 1000 people could be working on it at the same time. Is that number counting people working on the ops stuff or building other tooling not part of the main app?

Think of it this way. 100 different projects. Each with different business goals etc. Make some of the teams Platform teams for good measure.

Re: Successfully Merging the Work of 1000 Developers

#105

Post by leetbulb » Thu, Nov 21, 2019, 12:59 PM UTC

baroffoos wrote: post #94

irfansharif wrote: post #41

Earlier quoted context omitted.

The core shopify codebase is a monolithic one, hence the merging infrastructure described in the post. It's not a "single project" in any useful definition of the phrase.

Im interested how what looks to me like a fairly standard web store could have such an enormous amount if developers working on it. Is there some kind of insane complexity behind webstores I am not seeing here?

Shopify is far more than just a "fairly standard web store"

Re: Successfully Merging the Work of 1000 Developers

#106

Post by w-m » Thu, Nov 21, 2019, 2:02 PM UTC

jrockway wrote: post #38

w-m wrote: post #21

Earlier quoted context omitted.

Point 2 is very important and very hard to get right. For unit tests, there is a clear dependency on the code and you can easily just run a subset of the tests. But wouldn't you have to run any system and integration tests of the affected module, as it's not clear what effects the code change can have? This will blow up CI times again. How did Google deal with this?

Your integration test needed the system that you were integrating with, so you'd have to declare that as a dependency. My philosophy was to always have integration tests run in the normal CI system. This basically meant creating a test binary that happened to link in the systems you were integrating with, and run tests against that. This is easier when everything is written in the same programming language, and for t…

Wow, thanks for the detailed reply. You mentioned a couple of implementations that I hadn't thought about. But I guess the short version would be, as so often: testing systems is hard, and there's no one-fits-all solution.

Re: Successfully Merging the Work of 1000 Developers

#107

Post by jefurii » Thu, Nov 21, 2019, 7:01 PM UTC

This seems like a complicated way of shoehorning the Linux kernel mailing list workflow into GitHub.

Re: Successfully Merging the Work of 1000 Developers

#108

Post by strbean » Thu, Nov 21, 2019, 9:36 PM UTC

jrockway wrote: post #38

w-m wrote: post #21

Earlier quoted context omitted.

Point 2 is very important and very hard to get right. For unit tests, there is a clear dependency on the code and you can easily just run a subset of the tests. But wouldn't you have to run any system and integration tests of the affected module, as it's not clear what effects the code change can have? This will blow up CI times again. How did Google deal with this?

Your integration test needed the system that you were integrating with, so you'd have to declare that as a dependency. My philosophy was to always have integration tests run in the normal CI system. This basically meant creating a test binary that happened to link in the systems you were integrating with, and run tests against that. This is easier when everything is written in the same programming language, and for t…

Tangentially:

I've seen several blog posts from Google about using fakes and 'hermetic servers' for testing. We use GCP for our product, and unfortunately, Google doesn't seem to care much about making this easy. For example, I think I saw only one or two languages for which the Google Storage client libraries provided "fakes" of a Google Storage server. For PubSub (and maybe one or two other services?) there is the PubSub Emulator, which is unfortunately in Java and isn't supported by any of the CLI tools.

For all their love of fakes and hermetic servers, it would be awesome if they provided them for all the GCP services.