Keeping master green at scale • Hacker News Board

Re: Keeping master green at scale

#71

Post by sundargates » Thu, Apr 18, 2019, 9:55 PM UTC

huac wrote: post #37

pastor_elm wrote: post #20

Earlier quoted context omitted.

Sounds so much simpler outside the context of a 'research' paper: >When an engineer attempts to land their commit, it gets enqueued on the Submit Queue. This system takes one commit at a time, rebases it against master, builds the code and runs the unit tests. If nothing breaks, it then gets merged into master. With Submit Queue in place, our master success rate jumped to 99%. https://eng.uber.com/ios-monorepo/

submit queue makes sense and is used by lots of people, it's the "machine learning" which is applied to choosing commits to enqueue which I found to be interesting. if the master success rate was already 99% in 2017, with just submit queue, why build the complex ML stuff?

The way we achieved the master success rate of 100% at scale was by using the techniques that we describe in the paper. The blog doesn't go into details on Submit Queue and how it works.

Just to clarify, the ML models are used to predict the prob. that a given change will succeed against master as well as the prob. of conflict between changes.

Re: Keeping master green at scale

#72

Post by sundargates » Thu, Apr 18, 2019, 10:05 PM UTC

mcqueenjordan wrote: post #15

huac wrote: post #9

"Based on all possible outcomes of pending changes, SubmitQueue constructs, and continuously updates a speculation graph that uses a probabilistic model, powered by logistic regression. The speculation graph allows SubmitQueue to select builds that are most likely to succeed, and speculatively execute them in parallel" This is either brilliant or just something built for a promotion packet

Promotion-oriented design, no doubt.

I can guarantee you that none of the ideas on the paper were born out of a desire to get promoted. They were invented because ML models helped figure out which set of builds we need to run more accurately at scale.

Re: Keeping master green at scale

#73

Post by sundargates » Thu, Apr 18, 2019, 10:13 PM UTC

richardwhiuk wrote: post #22

Anyone fancy comparing this to bors?

Actually we have compared it in our paper.

Bors builds one change at a time. On the other hand, Submit Queue speculatively builds several changes at a time based on the outcomes of other pending changes in the system. Apart from that, Submit Queue uses a conflict analyzer to find independent changes in order to commit changes in parallel as well as trim the speculation graph.

We have also evaluated the performance of Single-Queue (idea of Bors) on our workloads. In fact, as described in the paper, the performance of this technique at scale was so high (~132x slower) that we omitted its results. Submit Queue on the other hand operates at 1-3x region compared to an optimal solution.

I recommend you to read the paper here for further details. https://dl.acm.org/citation.cfm?id=3303970

Re: Keeping master green at scale

#74

Post by filoleg » Thu, Apr 18, 2019, 10:35 PM UTC

vruiz wrote: post #50

YawningAngel wrote: post #48

Earlier quoted context omitted.

Most places I know of use Git Virtual File System or equivalents.

It is my understanding that VFSForGit only works on Windows.

The github repo has instructions for running it on Mac and says that the stable Mac version is under active development.

Re: Keeping master green at scale

#75

Post by tomschlick » Thu, Apr 18, 2019, 10:43 PM UTC

viklove wrote: post #58

roskilli wrote: post #55

Earlier quoted context omitted.

I don't want to come across as negative, but just an observation and to play devil's advocate - wouldn't it be better to fix the flaky test or delete it entirely instead of build a feature to disable it during a test run in an automated fashion? Whenever our team has a significant number of flakey tests (more than 1-2) we usually schedule a bug squash session to fix them and amortize the cost over the whole team.

Best practice is actually just to disable all tests that are failing. Can't hold up our sprint deadlines!

Failing != flaking. If your tests interact with any level of randomness (seed data, time based constraints, etc) you're going to find the occasional test that doesn't work and subsequently works on the rebuild.

If something is consistently failing I would assume this tool does not disable it.

Re: Keeping master green at scale

#76

Post by nevir » Thu, Apr 18, 2019, 10:56 PM UTC

Aqua_Geek wrote: post #25

googlemike wrote: post #8

Earlier quoted context omitted.

I like it at Google!

Google has the team + tooling to properly support it. The same cannot be said for many other orgs.

Many teams

Re: Keeping master green at scale

#77

Post by xiphias2 » Thu, Apr 18, 2019, 11:00 PM UTC

sundargates wrote: post #72

mcqueenjordan wrote: post #15

Earlier quoted context omitted.

Promotion-oriented design, no doubt.

I can guarantee you that none of the ideas on the paper were born out of a desire to get promoted. They were invented because ML models helped figure out which set of builds we need to run more accurately at scale.

I still hope you got promoted though, you deserve it :)

Re: Keeping master green at scale

#78

Post by draw_down » Thu, Apr 18, 2019, 11:09 PM UTC

state_less wrote: post #38

pastor_elm wrote: post #20

Earlier quoted context omitted.

Sounds so much simpler outside the context of a 'research' paper: >When an engineer attempts to land their commit, it gets enqueued on the Submit Queue. This system takes one commit at a time, rebases it against master, builds the code and runs the unit tests. If nothing breaks, it then gets merged into master. With Submit Queue in place, our master success rate jumped to 99%. https://eng.uber.com/ios-monorepo/

You can get most of the benefit on smaller scales by building feature branches and ensuring they pass unit tests, deployment and integration testing before they're allowed to be merged to mainline. It still depends on well written tests, lest your confidence be dashed when a human starts pushing buttons and pulling levers. Also, don't break up tightly coupled code/modules into separate repos for the sake of microserv…

It isn't just that the tests need to be good and humans break things sometimes. At a certain scale, the following happens enough to be a problem:

- changeset A is submitted, an integration branch is cut from latest master, and CI begins

- changeset B is submitted, an integration branch is cut from latest master, and CI begins

- changeset A's integration branch passes CI build/test, so A is merged into master

- changeset B's integration branch passes CI build/test, so B is merged into master

- however, changeset A + B interact in such a way that causes build and/or tests to fail

- build is now broken

You're probably thinking "that sounds like it wouldn't happen very often. Both changes would need to be submitted within some window such that changeset B's integration branch does not include changeset A, and vice-versa". Which is correct, but that's where the scale comes in. With enough engineers this starts happening more, and the more engineers you have the more unacceptable it is to have the build broken for any amount of time. And the more engineers the more code you have so the longer any individual build starts taking which lengthens the window during which the two conflicting changes could be submitted.

You need to do it in a way that serializes the changes because that's the only way to prevent this, but that takes too long. So the paper is about how to solve this problem.

Re: Keeping master green at scale

#79

Post by zb » Thu, Apr 18, 2019, 11:18 PM UTC

ori_b wrote: post #21

underrun wrote: post #19

Adrian Colyer dug into this a little further on the morning paper: https://blog.acolyer.org/2019/04/18/keeping-master-green-at-... His analysis indicates that what uber does as part of its build pipeline is to break up the monorepo into "targets" and for each target create something like a merkle tree (which is basically what git uses to represent commits) and use that information to detect potential conflicts (for m…

> For multirepo users this is explicit in that this comes for free :-) Only if you spend the time to build tools to detect commits in your dependencies, as well as your dependent repositories, and figure out how to update and check them out on the appropriate builds. So, no, it doesn't come for free.

Not exactly for free, but there are free tools that handle this job for you very nicely:

https://zuul-ci.org/

Re: Keeping master green at scale

#80

Post by dilyevsky » Thu, Apr 18, 2019, 11:38 PM UTC

antimora wrote: post #66

I am still trying to wrap my head around a giant monolithic repo model instead of breaking codes into multiple repos. At Amazon, for example, they have multi repos setup. A single repo represents one package which has major version.The Amazon's build system builds packages and pulls dependencies from the artifact repository when needed. The build system is responsible for "what" to build vs "how" to build, which is l…

Monorepos are really nice if you want to enforce consistent and sane engineering practices and not waste time managing all the repos individually by teams.

Bazel has target caching including remote caching which can be shared across multiple engineers/execution environments. The tricky part would be ensuring your builds are hermetic and reproducible (which is also easier to achieve in monorepo setup).