Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keeping master green at scale

eng.uber.com

Post reply on HN
Print view View original

41–50 of 120 posts

Re: Keeping master green at scale

#41

Post by jacobparker »

underrun wrote: post #39
ori_b wrote: post #21

Earlier quoted context omitted.

> For multirepo users this is explicit in that this comes for free :-) Only if you spend the time to build tools to detect commits in your dependencies, as well as your dependent repositories, and figure out how to update and check them out on the appropriate builds. So, no, it doesn't come for free.

sorry, "this" is rather ambiguous. You are totally correct that to achieve the same performance, correctness, and overall master level "green"ness in a multirepo system you would have to either define or detect dependencies and dependent repos, build the entire affected chain, and test the result. That part is much easier in monorepo. What I was referring to with "this" is that Uber's method of detecting potential co…

> You are totally correct that to achieve the same performance, correctness, and overall master level "green"ness in a multirepo system you would have to either define or detect dependencies and dependent repos, build the entire affected chain, and test the result. That part is much easier in monorepo.

You also would need to do that as an atomic operation (in the mono-repo + especially with a commit queue you're building on the atomicity of git).

Having to unwinding that transaction if you aren't atomic can get you into a big mess at larger scales.

Here's a good related talk: C++ as a "Live at Head" language: https://www.youtube.com/watch?v=tISy7EJQPzI by Titus Winters (from Google).

Re: Keeping master green at scale

#42

Post by vruiz »

huac wrote: post #11
chairleader wrote: post #3

Quite a premise: "Giant monolithic source-code repositories are one of the fundamental pillars of the back end infrastructure in large and fast-paced software companies."

facebook, google, airbnb, quora, many more all use monorepo obviously there are many others who do not use monorepo (amazon comes to mind) but it's reasonable to claim that they are actually widely used and fundamental when used

Does anybody know how these companies development environments look like? I know about Piper at Google but how do the rest manage? Does every single engineer have the entire monorepo in their machines?

Re: Keeping master green at scale

#43

Post by joshuamorton »

pastor_elm wrote: post #20
huac wrote: post #9

"Based on all possible outcomes of pending changes, SubmitQueue constructs, and continuously updates a speculation graph that uses a probabilistic model, powered by logistic regression. The speculation graph allows SubmitQueue to select builds that are most likely to succeed, and speculatively execute them in parallel" This is either brilliant or just something built for a promotion packet

Sounds so much simpler outside the context of a 'research' paper: >When an engineer attempts to land their commit, it gets enqueued on the Submit Queue. This system takes one commit at a time, rebases it against master, builds the code and runs the unit tests. If nothing breaks, it then gets merged into master. With Submit Queue in place, our master success rate jumped to 99%. https://eng.uber.com/ios-monorepo/

That's because the research-y part is "how do we pick which commit to enqueue next", and that's harder to answer succinctly.

Re: Keeping master green at scale

#44

Post by ryanmarsh »

underrun wrote: post #19

Adrian Colyer dug into this a little further on the morning paper: https://blog.acolyer.org/2019/04/18/keeping-master-green-at-... His analysis indicates that what uber does as part of its build pipeline is to break up the monorepo into "targets" and for each target create something like a merkle tree (which is basically what git uses to represent commits) and use that information to detect potential conflicts (for m…

Funny that you would draw a comparison to a Merkle tree. At one client they had such coupling between systems CI/CD was nearly impossible without either an explosion of test environments or grinding everything to a near halt.

We began working with the idea of consensus based CI/CD. If you pushed a change, you published that to the network. It gave other systems the opportunity to run their full suite of tests against the deployment of your code. Some number of confirmations from dependent systems was required to consider your code "stable". This progressed nearly sequentially assembling something like a block chain.

Ultimately the client was unable to pull this off for the same reason they were unable to decouple the systems: lack of software engineering capability.

Re: Keeping master green at scale

#45

Post by ljm »

ricardobeat wrote: post #32
msangi wrote: post #24

Earlier quoted context omitted.

Package managers solve it quite well. Just depend on the latest version of your dependencies and tag a new version whenever they change.

This doesn’t work when an underlying system changes, and upgrading is mandatory for all clients or package dependants (happens often at scale for a multitude of reasons).

This is essentially the problem go has had, precisely because you could only ever pull the latest from master.

Introduce a breaking change into a common library and now you have to update every other dependency to support it.

Not so bad in a monorepo. But when your codebases are distributed?

Re: Keeping master green at scale

#46

Post by underrun »

jacobparker wrote: post #41
underrun wrote: post #39

Earlier quoted context omitted.

sorry, "this" is rather ambiguous. You are totally correct that to achieve the same performance, correctness, and overall master level "green"ness in a multirepo system you would have to either define or detect dependencies and dependent repos, build the entire affected chain, and test the result. That part is much easier in monorepo. What I was referring to with "this" is that Uber's method of detecting potential co…

> You are totally correct that to achieve the same performance, correctness, and overall master level "green"ness in a multirepo system you would have to either define or detect dependencies and dependent repos, build the entire affected chain, and test the result. That part is much easier in monorepo. You also would need to do that as an atomic operation (in the mono-repo + especially with a commit queue you're buil…

thanks for the link - looking forward to watching.

Currently, I'm not convinced that you need to track and apply commits across dependencies and dependent systems atomically/transactionally to have a sane build environment even at scale, but you definitely get that part free with monorepo.

Any links to docs or presentations that address that specific issue would be very welcome :-)

Re: Keeping master green at scale

#47

Post by k__ »

chairleader wrote: post #3

Quite a premise: "Giant monolithic source-code repositories are one of the fundamental pillars of the back end infrastructure in large and fast-paced software companies."

Yes, my first thought "I wish the systems I worked on in big corps were big monolithic giants"

Re: Keeping master green at scale

#48

Post by YawningAngel »

vruiz wrote: post #42
huac wrote: post #11

Earlier quoted context omitted.

facebook, google, airbnb, quora, many more all use monorepo obviously there are many others who do not use monorepo (amazon comes to mind) but it's reasonable to claim that they are actually widely used and fundamental when used

Does anybody know how these companies development environments look like? I know about Piper at Google but how do the rest manage? Does every single engineer have the entire monorepo in their machines?

Most places I know of use Git Virtual File System or equivalents.

Re: Keeping master green at scale

#49

Post by lallysingh »

huac wrote: post #37
pastor_elm wrote: post #20

Earlier quoted context omitted.

Sounds so much simpler outside the context of a 'research' paper: >When an engineer attempts to land their commit, it gets enqueued on the Submit Queue. This system takes one commit at a time, rebases it against master, builds the code and runs the unit tests. If nothing breaks, it then gets merged into master. With Submit Queue in place, our master success rate jumped to 99%. https://eng.uber.com/ios-monorepo/

submit queue makes sense and is used by lots of people, it's the "machine learning" which is applied to choosing commits to enqueue which I found to be interesting. if the master success rate was already 99% in 2017, with just submit queue, why build the complex ML stuff?

The Submit Queue is the name of the complex ML stuff.

Re: Keeping master green at scale

#50

Post by vruiz »

YawningAngel wrote: post #48
vruiz wrote: post #42

Earlier quoted context omitted.

Does anybody know how these companies development environments look like? I know about Piper at Google but how do the rest manage? Does every single engineer have the entire monorepo in their machines?

Most places I know of use Git Virtual File System or equivalents.

It is my understanding that VFSForGit only works on Windows.
Post reply on HN