Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Uber SubmitQueue: a high-performance speculative merge queue

github.com

Post reply on HN
Print view View original

11–20 of 63 posts

Re: Uber SubmitQueue: a high-performance speculative merge queue

#11

Post by dastbe »

kyrra wrote: post #8
mi_lk wrote: post #4

Monorepo benefits maybe a small set of developers that manages it for the whole company, at the cost of most other engineers who develop in it. Unnecessary waste of time. Case in point another new merge queue has to be created to somehow manage the messiness

It's really the case that Git is terrible at scale. Honestly most source control is. When you have teams of 10,000+ people working on code, if you wanted to shard it, it will be lots of little projects, then you're going to have dependency hell. The core in for teams will either have to spend most of their time upgrading everyone's packages, our other teams are going to have to deal with those upgrades. The core infr…

this was always the nice thing at amazon, because you never needed an atomic migration with version sets.

it was definitely a case of making many things worse to do than in a monorepo, but more things viable to do.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#12

Post by kccqzy »

Keeping your trunk consistently green at scale is probably too costly. Not even Google can keep its google3 monorepo consistently buildable let alone green. It’s more worthwhile to keep trunk mostly green, stop chasing the final 0.1%, and instead develop tooling to quickly identify culprits to be rolled back automatically.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#13

Post by inigyou »

kyrra wrote: post #8
mi_lk wrote: post #4

Monorepo benefits maybe a small set of developers that manages it for the whole company, at the cost of most other engineers who develop in it. Unnecessary waste of time. Case in point another new merge queue has to be created to somehow manage the messiness

It's really the case that Git is terrible at scale. Honestly most source control is. When you have teams of 10,000+ people working on code, if you wanted to shard it, it will be lots of little projects, then you're going to have dependency hell. The core in for teams will either have to spend most of their time upgrading everyone's packages, our other teams are going to have to deal with those upgrades. The core infr…

It should be sharded per team.

I think many shops made the mistake of sharding per deployment unit or compilation unit of a larger whole. My first exposure was sharding per executable file of a firmware image. That's just terrible. You can't coordinate changes that you want to be atomic in the firmware image.

In response I think some people went to monorepo to get atomic change sets but now they're too big. If you have an android app team and a backend team, and they don't deploy together or build together or do anything together besides agree on an API, which has to be backwards compatible anyway because apps don't instantly self-update, there's no need to put those things together if it causes headaches. It's probably better to have too many things together than too few, but neither is ideal.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#14

Post by AlotOfReading »

inigyou wrote: post #5
conradludgate wrote: post #3

Earlier quoted context omitted.

In my experience, it's a monorepo problem. We have a monorepo at work and the CI only runs a minority of tests prior to merging, and then reverts bad commits later. It can take a few hours until every test has run including your commit. I don't know enough about the setup to know how it decides to run what tests though

Yeah that's true. It becomes O(N^2) with one factor of N being the number of commits to the repo, and another factor being the amount of CI stuff you do every time the repo changes.

You can do better than O(N^2) if your changes are monotonic /independent (or you're able to cache for basically the same reason). This usually requires either luck, a huge amount of programmer discipline, or very different programming languages than we're used to like unison.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#15

Post by esafak »

esprehn wrote: post #7

Airbnb has a version of this internally that was pretty awesome called Evergreen, based on the Uber paper as well. I wish more companies would open source their monorepo infra. A well done monorepo is a huge force multiplier on a large organization, but the OSS world is lacking a lot of the infra so everyone starts from a painful place and works up or has a bad impression of monorepos. Google's Piper is another examp…

Keeping Master Green at Scale https://dl.acm.org/doi/10.1145/3302424.3303970

Re: Uber SubmitQueue: a high-performance speculative merge queue

#16

Post by medellin »

mi_lk wrote: post #4

Monorepo benefits maybe a small set of developers that manages it for the whole company, at the cost of most other engineers who develop in it. Unnecessary waste of time. Case in point another new merge queue has to be created to somehow manage the messiness

I know what kind of engineer you are from this comment.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#17

Post by kyrra »

esprehn wrote: post #7

Airbnb has a version of this internally that was pretty awesome called Evergreen, based on the Uber paper as well. I wish more companies would open source their monorepo infra. A well done monorepo is a huge force multiplier on a large organization, but the OSS world is lacking a lot of the infra so everyone starts from a painful place and works up or has a bad impression of monorepos. Google's Piper is another examp…

The problem with many of these tools is they are built on lots of other tech, like S3 dev called out: https://x.com/haipingfu/status/2084880266858995990

Piper is built on lots of other tech like spanner and chubby and many others. And some of those have specific hardware requirements that only exist in Google datacenters. Untangling that web of tech is neigh impossible (or not something worth the cost to leadership).

Re: Uber SubmitQueue: a high-performance speculative merge queue

#18

Post by jedberg »

I believe that the solution to the coordination problem is good monorepo tooling (like the OP) plus AI to understand the whole codebase and help the engineers understand how their part fits in.

I was one of the biggest proponents of microservices, going so far as to traveling around the world spreading the gospel of microservices keynoting large tech conferences. I believed that microservices were the best solution to scaling large teams of developers, so that small teams could work on small problems, where the API was the only contract between them.

But even then I cautioned that the overhead made it not worthwhile for small teams -- that it was a solution to the coordination problem for large organizations. And that Google was not a counterexample because they had spent so many resources on their monorepo tooling.

But there is a new factor in town that changes the calculus:

AIs can grok monorepos much easier than a cluster of microservices. AIs change the calculus here. They allow the developer to work successfully in even the largest monorepos, and the AIs themselves will give better results when all of the code is in one place.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#20

Post by fweimer »

conradludgate wrote: post #3
inigyou wrote: post #2

So they reinvented OpenStack's Zuul? I've always thought if you're pushing changes at the sort of velocity that requires it, you're doing something wrong, anyway.

In my experience, it's a monorepo problem. We have a monorepo at work and the CI only runs a minority of tests prior to merging, and then reverts bad commits later. It can take a few hours until every test has run including your commit. I don't know enough about the setup to know how it decides to run what tests though

It's not really about a monorepo, but how many changes you can have in flight in parallel. Desirability of merge-and-rebase-and-check depends on more factors: the rate of change, the number of developers, the time the pre-merge checks take and the reliability of those checks, and how often pre-merge checks fail (legitimately or otherwise). How conflict-prone queue changes are matters, too.

I would like to have something like this for one of the repositories I work on, which is pretty far away from being a monorepo (it's mainly producing two tightly interlinked binary objects). Fortunately, I believe Gitlab offers something like it, at least as a preview feature (under the namemerge-and-rebase or something like that).

Post reply on HN