Keeping master green at scale

#1

Post by roshanj » Thu, Apr 18, 2019, 5:16 PM UTC

Keeping master green at scale
eng.uber.com

Re: Keeping master green at scale

#2

Post by 7e » Thu, Apr 18, 2019, 5:45 PM UTC

Is this novel? Other companies have had this for ages.

Re: Keeping master green at scale

#3

Post by chairleader » Thu, Apr 18, 2019, 5:45 PM UTC

Quite a premise: "Giant monolithic source-code repositories are one of the fundamental pillars of the back end infrastructure in large and fast-paced software companies."

Re: Keeping master green at scale

#4

Post by ricardobeat » Thu, Apr 18, 2019, 6:05 PM UTC

7e wrote: post #2

Is this novel? Other companies have had this for ages.

No, they haven't. This is a system to queue commits, not a simple CI setup. This problem only comes up when you start having contention due to commit volume in a monorepo (think thousands commits/day). This is only the 3rd one I've heard about.

> This paper introduces a change management system called SubmitQueue that is responsible for continuous integration of changes into the mainline at scale while always keeping the mainline green. Based on all possible outcomes of pending changes, SubmitQueue constructs, and continuously updates a speculation graph that uses a probabilistic model, powered by logistic regression. The speculation graph allows SubmitQueue to select builds that are most likely to succeed, and speculatively execute them in parallel. Our system also uses a scalable conflict analyzer that constructs a conflict graph among pending changes. The conflict graph is then used to (1) trim the speculation space to further improve the likelihood of using remaining speculations, and (2) determine independent changes that can commit in parallel

Re: Keeping master green at scale

#5

Post by paxys » Thu, Apr 18, 2019, 6:13 PM UTC

7e wrote: post #2

Is this novel? Other companies have had this for ages.

Which companies?

Re: Keeping master green at scale

#6

Post by zyang » Thu, Apr 18, 2019, 6:15 PM UTC

ricardobeat wrote: post #4

7e wrote: post #2

Is this novel? Other companies have had this for ages.

No, they haven't. This is a system to queue commits, not a simple CI setup. This problem only comes up when you start having contention due to commit volume in a monorepo (think thousands commits/day). This is only the 3rd one I've heard about. > This paper introduces a change management system called SubmitQueue that is responsible for continuous integration of changes into the mainline at scale while always keeping…

I don't quite understand the problem they are trying to solve. Is there so many change sets that they couldn't provision enough ci servers, hence the "speculation graph with probabilistic model"?

Re: Keeping master green at scale

#7

Post by draw_down » Thu, Apr 18, 2019, 6:17 PM UTC

chairleader wrote: post #3

Quite a premise: "Giant monolithic source-code repositories are one of the fundamental pillars of the back end infrastructure in large and fast-paced software companies."

I'm willing to agree with that premise.

Re: Keeping master green at scale

#8

Post by googlemike » Thu, Apr 18, 2019, 6:18 PM UTC

chairleader wrote: post #3

Quite a premise: "Giant monolithic source-code repositories are one of the fundamental pillars of the back end infrastructure in large and fast-paced software companies."

I like it at Google!

Re: Keeping master green at scale

#9

Post by huac » Thu, Apr 18, 2019, 6:23 PM UTC

"Based on all possible outcomes of pending changes, SubmitQueue constructs, and continuously updates a speculation graph that uses a probabilistic model, powered by logistic regression. The speculation graph allows SubmitQueue to select builds that are most likely to succeed, and speculatively execute them in parallel"

This is either brilliant or just something built for a promotion packet

Re: Keeping master green at scale

#10

Post by roshanj » Thu, Apr 18, 2019, 6:24 PM UTC

ricardobeat wrote: post #4

7e wrote: post #2

Is this novel? Other companies have had this for ages.

No, they haven't. This is a system to queue commits, not a simple CI setup. This problem only comes up when you start having contention due to commit volume in a monorepo (think thousands commits/day). This is only the 3rd one I've heard about. > This paper introduces a change management system called SubmitQueue that is responsible for continuous integration of changes into the mainline at scale while always keeping…

The problem also occurs if your CI Build + Test steps take a while to run, even on a small team pushing dozens of commits per day.

Two code-conflict-free changes may pass a pre-merge build+test cycle independently but may logically break one another if both changes are merged into master. Using a submit/merge queue guarantees that each change has passed tests with the exact ordering of commits it would be merged onto. The example described here is a better explanation: https://github.com/bors-ng/bors-ng#but-dont-githubs-protecte...