Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Successfully Merging the Work of 1000 Developers

engineering.shopify.com

Post reply on HN
Print view View original

91–100 of 108 posts

Re: Successfully Merging the Work of 1000 Developers

#91

Post by aledalgrande »

Very surprised they hadn't locked merge to master until recently.

I do like the emergency exception. The problem then becomes having a solid policy on when to use it.

Not really sure about the flakiness test (the 25% figure is totally random), as you could have a test that is failing only at a certain hour of the day and you would be furiously rerunning the test suite for nothing because it would always fail, wasting resources. It would be much better to at least single out the failing tests and rerun only those, dropping the ones that pass successfully until max tries are reached or the queue is empty.

Re: Successfully Merging the Work of 1000 Developers

#92

Post by glennpratt »

jrockway wrote: post #25
sshumaker wrote: post #18

Earlier quoted context omitted.

Xoogler here. When I left in 2015 there were definitely teams that used merge queues (i.e. TAP presubmit). Generally these were teams with a more monolithic architecture, like YouTube that had a massive Python mono.

I guess TAP presubmit might be a merge queue... but it seems different from this. There was no requirement that some mechanical system checked that tests passed before your merged your CL. You could merge any code whenever it was approved. If you felt like running the tests, good for you. TAP presubmit is just that mechanical system that runs your tests before executing the merge. That seems like traditional CI to me…

I have never seen anyone automate what Bors does with Jenkins and have anything approaching decent UX. The closest I've ever seen is a permanent stage branch that sometimes has automatic promotion, little integration with reviews and inevitably breaks every few weeks until some poor soul debugs it.

Re: Successfully Merging the Work of 1000 Developers

#93

Post by robocat »

peterwwillis wrote: post #78
robocat wrote: post #71

Earlier quoted context omitted.

Google don't solve it your way: https://news.ycombinator.com/item?id=21586180

Yes, that's correct, Google invented its own proprietary distributed object store and distributed version control system and distributed Linux-only filesystem and distributed build-and-test-system to work with a single SDLC that its entire company must follow strictly to release anything, just so it could keep using a single repository. What's your point?

Clearly given those costs, Google really believe in mono-repo, and presumably they have tried to back it up with internal stats?

Although hard to get stats without control group - maybe control group could be acquisitions?

Re: Successfully Merging the Work of 1000 Developers

#94

Post by baroffoos »

irfansharif wrote: post #41
coldcode wrote: post #30

I find it difficult to imagine why you would need 1000 developers in most single projects. The cost of managing so many people and work streams would seem to overwhelm whatever volume you could get out of them

The core shopify codebase is a monolithic one, hence the merging infrastructure described in the post. It's not a "single project" in any useful definition of the phrase.

Im interested how what looks to me like a fairly standard web store could have such an enormous amount if developers working on it. Is there some kind of insane complexity behind webstores I am not seeing here?

Re: Successfully Merging the Work of 1000 Developers

#95

Post by evan_ »

coldcode wrote: post #30

I find it difficult to imagine why you would need 1000 developers in most single projects. The cost of managing so many people and work streams would seem to overwhelm whatever volume you could get out of them

When you have 100 devs working on the same codebase the bugs start to get so gnarly you need 900 more devs to fix them.

Re: Successfully Merging the Work of 1000 Developers

#96

Post by jacktli »

baroffoos wrote: post #94
irfansharif wrote: post #41

Earlier quoted context omitted.

The core shopify codebase is a monolithic one, hence the merging infrastructure described in the post. It's not a "single project" in any useful definition of the phrase.

Im interested how what looks to me like a fairly standard web store could have such an enormous amount if developers working on it. Is there some kind of insane complexity behind webstores I am not seeing here?

It’s about providing a platform for enabling global commerce. Our engineering blog has other posts that goes into technical depth about a lot of the challenges we face, and our solutions. For context, here is some info about Black Friday/Cyber Monday for us last year: https://engineering.shopify.com/blogs/engineering/preparing-...

Re: Successfully Merging the Work of 1000 Developers

#97

Post by clintonb »

ljm wrote: post #67
demosito666 wrote: post #66

Earlier quoted context omitted.

I've read the description, but I fail to understand how this is different from just merging current master into the PR and running integration on that before merging in back into master? This can be done with 10 lines of groovy in Jenkins. Also the exact workflow described with staging branch and batch merge is probably another 20-30 lines.

This is what Travis CI did/does and I have not seen another CI platform offer the same. Travis runs your test suite on the merge commit, not the head of the branch.

Travis tests both the development branch and the merge/integration branch.

https://docs.travis-ci.com/user/pull-requests/#double-builds...

Re: Successfully Merging the Work of 1000 Developers

#98

Post by baroffoos »

jacktli wrote: post #96
baroffoos wrote: post #94

Earlier quoted context omitted.

Im interested how what looks to me like a fairly standard web store could have such an enormous amount if developers working on it. Is there some kind of insane complexity behind webstores I am not seeing here?

It’s about providing a platform for enabling global commerce. Our engineering blog has other posts that goes into technical depth about a lot of the challenges we face, and our solutions. For context, here is some info about Black Friday/Cyber Monday for us last year: https://engineering.shopify.com/blogs/engineering/preparing-...

I understand its a hugely popular service and there is a lot that goes in to something so big but there are bigger websites that run on much much smaller teams of developers. I just don't understand how there is even enough surface area on the app that 1000 people could be working on it at the same time. Is that number counting people working on the ops stuff or building other tooling not part of the main app?

Re: Successfully Merging the Work of 1000 Developers

#99

Post by dudzik »

dustym wrote: post #86

Anyone know what build times are for the Shopify monolith?

I'm working on the test-infra team at Shopify. We got the CI down in the last couple of weeks to around 22m 50th percentile and 33m 95th percentile. At the moment we get most of our speed up by parallizing our builds steps a lot. But we hit a ceiling with that and are working on a project on selectively running tests.
Post reply on HN