Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keeping master green at scale

eng.uber.com

Post reply on HN
Print view View original

21–30 of 120 posts

Re: Keeping master green at scale

#21

Post by ori_b »

underrun wrote: post #19

Adrian Colyer dug into this a little further on the morning paper: https://blog.acolyer.org/2019/04/18/keeping-master-green-at-... His analysis indicates that what uber does as part of its build pipeline is to break up the monorepo into "targets" and for each target create something like a merkle tree (which is basically what git uses to represent commits) and use that information to detect potential conflicts (for m…

> For multirepo users this is explicit in that this comes for free :-)

Only if you spend the time to build tools to detect commits in your dependencies, as well as your dependent repositories, and figure out how to update and check them out on the appropriate builds.

So, no, it doesn't come for free.

Re: Keeping master green at scale

#23

Post by jrochkind1 »

joshuamorton wrote: post #14
zyang wrote: post #6

Earlier quoted context omitted.

I don't quite understand the problem they are trying to solve. Is there so many change sets that they couldn't provision enough ci servers, hence the "speculation graph with probabilistic model"?

Sort of, though not really. Imagine I have three changes, C1 modifies F1, C2 modifies F2, and C3 modifies F1. There's no relation between F1 and F2. At low-ish rate of submission, you test and commit C1, then test and commit C2, then when you try and test and commit C3, you rebase, and re-test and commit. (the merge doesn't conflict so can be automatically fixed) Now assume all three changes are submitted by 3 differ…

awesome explanation

Re: Keeping master green at scale

#24

Post by msangi »

ori_b wrote: post #21
underrun wrote: post #19

Adrian Colyer dug into this a little further on the morning paper: https://blog.acolyer.org/2019/04/18/keeping-master-green-at-... His analysis indicates that what uber does as part of its build pipeline is to break up the monorepo into "targets" and for each target create something like a merkle tree (which is basically what git uses to represent commits) and use that information to detect potential conflicts (for m…

> For multirepo users this is explicit in that this comes for free :-) Only if you spend the time to build tools to detect commits in your dependencies, as well as your dependent repositories, and figure out how to update and check them out on the appropriate builds. So, no, it doesn't come for free.

Package managers solve it quite well. Just depend on the latest version of your dependencies and tag a new version whenever they change.

Re: Keeping master green at scale

#25

Post by Aqua_Geek »

googlemike wrote: post #8
chairleader wrote: post #3

Quite a premise: "Giant monolithic source-code repositories are one of the fundamental pillars of the back end infrastructure in large and fast-paced software companies."

I like it at Google!

Google has the team + tooling to properly support it. The same cannot be said for many other orgs.

Re: Keeping master green at scale

#26

Post by ratRaces »

This isn't really news. Managing a technical code base is mostly about excluding total fucking dipshits.

Excluding total fucking dipshits is often about hiring smart people, and not exterminating their enthusiasm and optimism, so that their willingness to perform doesn't wither and die in a smothering rat race environment.

Re: Keeping master green at scale

#27

Post by msangi »

huac wrote: post #11
chairleader wrote: post #3

Quite a premise: "Giant monolithic source-code repositories are one of the fundamental pillars of the back end infrastructure in large and fast-paced software companies."

facebook, google, airbnb, quora, many more all use monorepo obviously there are many others who do not use monorepo (amazon comes to mind) but it's reasonable to claim that they are actually widely used and fundamental when used

All the companies have in common a huge budget they can invest on their build systems to overcome the shortcomings of monorepos.

They do have some benefits, but they also come with an immense cost

Re: Keeping master green at scale

#28

Post by ratRaces »

huac wrote: post #11
chairleader wrote: post #3

Quite a premise: "Giant monolithic source-code repositories are one of the fundamental pillars of the back end infrastructure in large and fast-paced software companies."

facebook, google, airbnb, quora, many more all use monorepo obviously there are many others who do not use monorepo (amazon comes to mind) but it's reasonable to claim that they are actually widely used and fundamental when used

A monorepo isn't particularly innovative, if the technical subgroups can create arbitrary subdirectories and branch ad infinitum.

Of course it's easy to work with a monorepo, when you can create a folder to stash your stuff in, bury important stuff deep in a nested tree under your home root, and then hide iterations in branches you own.

That's what every smart person does in repos they have no control over.

Re: Keeping master green at scale

#29

Post by jade12 »

revskill wrote: post #16

What's exactly a monothlic ? Is it only related to codebase (monothlic vs monorepo) ? Or it's about runtime like microservices vs monothlic.

From the first sentence of the abstract:

> monolithic source-code repositories

A monorepo is a monolithic repository

Re: Keeping master green at scale

#30

Post by patejam »

msangi wrote: post #27
huac wrote: post #11

Earlier quoted context omitted.

facebook, google, airbnb, quora, many more all use monorepo obviously there are many others who do not use monorepo (amazon comes to mind) but it's reasonable to claim that they are actually widely used and fundamental when used

All the companies have in common a huge budget they can invest on their build systems to overcome the shortcomings of monorepos. They do have some benefits, but they also come with an immense cost

But that's what the OP's quote is saying. "large" companies use them.
Post reply on HN