Keeping master green at scale • Hacker News Board

Re: Keeping master green at scale

#101

Post by joatmon-snoo » Fri, Apr 19, 2019, 7:08 AM UTC

underrun wrote: post #39

ori_b wrote: post #21

Earlier quoted context omitted.

> For multirepo users this is explicit in that this comes for free :-) Only if you spend the time to build tools to detect commits in your dependencies, as well as your dependent repositories, and figure out how to update and check them out on the appropriate builds. So, no, it doesn't come for free.

sorry, "this" is rather ambiguous. You are totally correct that to achieve the same performance, correctness, and overall master level "green"ness in a multirepo system you would have to either define or detect dependencies and dependent repos, build the entire affected chain, and test the result. That part is much easier in monorepo. What I was referring to with "this" is that Uber's method of detecting potential co…

> If Bob commits to repo A and Sally commits to repo B, their commits can't result in a merge conflict.

This holds true when A and B are leaf repos, but gets tricky with repos inside a dependency graph. More concretely, if C depends on both A and B, and it turns out that C depends on A and B in such a way that A_bob and B_sally are mutually incompatible, you need some kind of mechanism for reconciling that.

Of course, exactly as you point out, mono and multi are two tradeoffs for the problems that large codebases intrinsically are.

Re: Keeping master green at scale

#102

Post by bjackman » Fri, Apr 19, 2019, 8:09 AM UTC

viraptor wrote: post #90

Scaevolus wrote: post #62

There's a nice middle ground between this and a one-at-a-time submit queue: have a speculative batch running on the side. This gives nice speedups (approaching N times more commits, where N is the batch size) with minimal complexity. One useful metric is the ratio between test time and the number of commits per day. If your tests run in a minute, you can test submissions one at a time and still have a thousand succes…

The same thing was (is?) done in openstack with zuul, I believe. When you going to merge something, your branch goes on top of things already going through the CI.

Yes, I recently switched my org to using Zuul for this purpose; by having an internal speculative queue of future states for master you can have multiple pending changes tested at once, while also ensuring that the tested code is exactly what goes into master. So far it's been a really good experience, in particular as our tests take a long time.

It sounds like Uber's thing has a lot more smarts regardint deciding what gets tested. For the scale I work at (<200k lines of code) that isn't necessary.

Re: Keeping master green at scale

#103

Post by ricardobeat » Fri, Apr 19, 2019, 9:40 AM UTC

erik_seaberg wrote: post #95

ricardobeat wrote: post #32

Earlier quoted context omitted.

This doesn’t work when an underlying system changes, and upgrading is mandatory for all clients or package dependants (happens often at scale for a multitude of reasons).

That's not good stewardship. You have a better API? Great, convince us it's worth investing in soon, you can even deprecate the known-good version. There's always a window where both will be in use, because we can't synchronously replace every running process everywhere (not that it's even a good idea without a canary). The shorter you try to make that window, the more needless pain is created and plans disrupted. Wh…

That's not reality for most large companies, even though it's the right mindset for most software libraries. Ex: A new legal requirement comes in, resulting in a mandate that fields X and Y for certain queries, that are being done all over the codebase, now have to be tokenized. This is a breaking and mandatory change, with no room to allow systems to stay behind, and expensive consequences.

In this case you'll have a very short transition, between all consumers updating their client code (possibly in a backwards compatible way) and the change in the implicated system being deployed, not the other way around.

Re: Keeping master green at scale

#104

Post by twic » Fri, Apr 19, 2019, 10:23 AM UTC

huac wrote: post #11

chairleader wrote: post #3

Quite a premise: "Giant monolithic source-code repositories are one of the fundamental pillars of the back end infrastructure in large and fast-paced software companies."

facebook, google, airbnb, quora, many more all use monorepo obviously there are many others who do not use monorepo (amazon comes to mind) but it's reasonable to claim that they are actually widely used and fundamental when used

Notably, Netflix doesn't (or least didn't):

https://medium.com/netflix-techblog/towards-true-continuous-...

Re: Keeping master green at scale

#105

Post by jeletonskelly » Fri, Apr 19, 2019, 1:04 PM UTC

ljm wrote: post #45

ricardobeat wrote: post #32

Earlier quoted context omitted.

This doesn’t work when an underlying system changes, and upgrading is mandatory for all clients or package dependants (happens often at scale for a multitude of reasons).

This is essentially the problem go has had, precisely because you could only ever pull the latest from master. Introduce a breaking change into a common library and now you have to update every other dependency to support it. Not so bad in a monorepo. But when your codebases are distributed?

It's almost like semantic versioning exists for some reason

Re: Keeping master green at scale

#106

Post by Shish2k » Fri, Apr 19, 2019, 1:10 PM UTC

vruiz wrote: post #42

huac wrote: post #11

Earlier quoted context omitted.

facebook, google, airbnb, quora, many more all use monorepo obviously there are many others who do not use monorepo (amazon comes to mind) but it's reasonable to claim that they are actually widely used and fundamental when used

Does anybody know how these companies development environments look like? I know about Piper at Google but how do the rest manage? Does every single engineer have the entire monorepo in their machines?

At facebook, a virtual filesystem (https://github.com/facebookexperimental/eden) + change monitor (https://facebook.github.io/watchman/) = source control operations run in O(files I have modified) instead of O(size of repository) time

Re: Keeping master green at scale

#107

Post by vruiz » Fri, Apr 19, 2019, 2:10 PM UTC

Shish2k wrote: post #106

vruiz wrote: post #42

Earlier quoted context omitted.

Does anybody know how these companies development environments look like? I know about Piper at Google but how do the rest manage? Does every single engineer have the entire monorepo in their machines?

At facebook, a virtual filesystem ( https://github.com/facebookexperimental/eden ) + change monitor ( https://facebook.github.io/watchman/ ) = source control operations run in O(files I have modified) instead of O(size of repository) time

Very interesting, thanks!

Re: Keeping master green at scale

#108

Post by Cthulhu_ » Fri, Apr 19, 2019, 2:18 PM UTC

state_less wrote: post #38

pastor_elm wrote: post #20

Earlier quoted context omitted.

Sounds so much simpler outside the context of a 'research' paper: >When an engineer attempts to land their commit, it gets enqueued on the Submit Queue. This system takes one commit at a time, rebases it against master, builds the code and runs the unit tests. If nothing breaks, it then gets merged into master. With Submit Queue in place, our master success rate jumped to 99%. https://eng.uber.com/ios-monorepo/

You can get most of the benefit on smaller scales by building feature branches and ensuring they pass unit tests, deployment and integration testing before they're allowed to be merged to mainline. It still depends on well written tests, lest your confidence be dashed when a human starts pushing buttons and pulling levers. Also, don't break up tightly coupled code/modules into separate repos for the sake of microserv…

This is what we did for a while at a project; there's options in most git hosts nowadays that force any PR to be up to date with master before merging on the one hand, and to have a green pipeline on the other. That works fine for smaller projects, but because it's not automated you end up with quite a bit of manual labor (rebase on master, push, wait for CI, discover someone else merged into master first, repeat).

Re: Keeping master green at scale

#109

Post by Cthulhu_ » Fri, Apr 19, 2019, 2:20 PM UTC

numlocked wrote: post #85

huac wrote: post #37

Earlier quoted context omitted.

submit queue makes sense and is used by lots of people, it's the "machine learning" which is applied to choosing commits to enqueue which I found to be interesting. if the master success rate was already 99% in 2017, with just submit queue, why build the complex ML stuff?

If there are 1000 commits per day (which wouldn't be that many), that's 10 master breaks per day.

And at least in my limited experience, the impact of master being broken is pretty big, and even bigger when you have multiple teams. Either you block master - leading to a lot less than those 1000 commits making it to master on that day - or continue merging in stuff, which causes the root cause of the master branch to become fuzzy - and if your reporting is not in order, that is, if the person who broke the build isn't told they did, it'll be a lot of "Who broke master?" and people looking at each other to find out who is going to look into it. That's not scalable.

Re: Keeping master green at scale

#110

Post by sundargates » Fri, Apr 19, 2019, 2:40 PM UTC

underrun wrote: post #82

sundargates wrote: post #69

Earlier quoted context omitted.

You are right to say that conflict analyzer tries to treat commits independently based on the service or app (which are usually in separate repositories in a multi-repo world). However, note that the problem of conflicting changes (or a red master) exists even when you are in a multi-repo world as you could have one repository getting a large number of commits. In fact, at Uber we have seen that behaviour with one of…

Do you mean the construct of probabilistic speculation applies in multirepo because you may end up with a hot spot repo that receives a high volume of commits at once? Or do you mean that multirepo could also benefit from the construct of probabilistic speculation by ordering commits across multiple repos such that you are maximizing the number of repos that have changed before you build and minimising the number of…

the former actually.

If you have 1 app that's the bread and butter of your company and, 60% of your 2000+ engineers working on various features of that one app, then even in a multi-repo world, you are going to have that 1 repo receiving ton of commits and the problem of keeping it green remains. Prob. speculation helps there.