Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Uber SubmitQueue: a high-performance speculative merge queue

github.com

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Uber SubmitQueue: a high-performance speculative merge queue

#31

Post by fcarraldo »

mupuff1234 wrote: post #28
esprehn wrote: post #7

Airbnb has a version of this internally that was pretty awesome called Evergreen, based on the Uber paper as well. I wish more companies would open source their monorepo infra. A well done monorepo is a huge force multiplier on a large organization, but the OSS world is lacking a lot of the infra so everyone starts from a painful place and works up or has a bad impression of monorepos. Google's Piper is another examp…

I don't understand how come monorepos never just got "solved", and why git didn't expand in that direction. I switched from a company with a monorepo to one without, and it just feels like going back to the stone age.

I work at a monorepo company and any time someone gets to work on a project that necessitates working outside of the monorepo, it's a night-and-day improvement.

Tools, especially open source ones (linters, static analysis, scanning, LSPs, IDEs, etc) are not built for monorepos, and with AI Agents working in the monorepo results in an enormous increase in input tokens as the agents are constantly trying to grep this giant source tree.

I'm sure it's possible that we're doing the monorepo thing wrong, but I'm genuinely curious what the upside is that you're experiencing? Or are these drawbacks unique to our implementation?

Re: Uber SubmitQueue: a high-performance speculative merge queue

#32

Post by jedberg »

consensus1 wrote: post #24
jedberg wrote: post #18

I believe that the solution to the coordination problem is good monorepo tooling (like the OP) plus AI to understand the whole codebase and help the engineers understand how their part fits in. I was one of the biggest proponents of microservices, going so far as to traveling around the world spreading the gospel of microservices keynoting large tech conferences. I believed that microservices were the best solution t…

Microservices is a deployment strategy. Monorepo is a code organization strategy. They are not mutually exclusive.

While you are technically correct (the best kind of correct), I would challenge you to find an organization that is doing the monorepo/microservice combo correctly. And what I mean by that is with microservices, the API is the only contract. Every service should be deployable independently.

If you have a monorepo, you're almost certainly violating that somewhere, using a shared library, or a shared database, or even just blocking deployment because some other service has turned the repo red.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#33

Post by fcarraldo »

jedberg wrote: post #18

I believe that the solution to the coordination problem is good monorepo tooling (like the OP) plus AI to understand the whole codebase and help the engineers understand how their part fits in. I was one of the biggest proponents of microservices, going so far as to traveling around the world spreading the gospel of microservices keynoting large tech conferences. I believed that microservices were the best solution t…

IME, AI does find established patterns more quickly a in monorepo (sometimes the ones you want, sometimes the ones you don't) - but at the cost of an enormous overhead tax you pay on input token cost.

Giving agents pointers to the right patterns, libraries and services helps avoid expensive grep goose chases, but if you're already curating the input you can do the same thing with small repositories.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#34

Post by mnahkies »

jedberg wrote: post #18

I believe that the solution to the coordination problem is good monorepo tooling (like the OP) plus AI to understand the whole codebase and help the engineers understand how their part fits in. I was one of the biggest proponents of microservices, going so far as to traveling around the world spreading the gospel of microservices keynoting large tech conferences. I believed that microservices were the best solution t…

I'm going to play devil's advocate here.

As a developer, I quite like monorepos to a certain size (eg: until they get big enough that the tooling we typically use outside of big tech starts to fall down).

As an AI, I'm not sure that I care? I'd guess that context management can actually be easier if each microservice has a well documented API (openapi/graphql/grpc/asyncapi/whatever) and you provide the agent harness the ability to drop into each polyrepo as required (and give it the ability to access said documentation).

The tedium of making branches / commits / pull requests across 6 repos to land a feature is less problematic to an agent.

Admittedly the way I'm using agents at the moment is more repo orientated where it's sandboxed to a single repo, but conceptually I think polyrepo microservices could end up being a sweet spot.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#35

Post by wasmperson »

Alternatively: don't worry about keeping trunk "green" at all. Have a second branch called "stable" or something that auto-fast-forwards to latest trunk whenever trunk is green. Checkout stable, push new changes to trunk, avoid breaking CI, but if you break CI then don't worry about it, just push up a fix.

If you act like trunk is this "sacred" thing that must always be ready to deploy then what you end up with is a bunch of long-lived branches and PRs and all the merge conflicts and overhead that come with those.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#36

Post by jedberg »

mnahkies wrote: post #34
jedberg wrote: post #18

I believe that the solution to the coordination problem is good monorepo tooling (like the OP) plus AI to understand the whole codebase and help the engineers understand how their part fits in. I was one of the biggest proponents of microservices, going so far as to traveling around the world spreading the gospel of microservices keynoting large tech conferences. I believed that microservices were the best solution t…

I'm going to play devil's advocate here. As a developer, I quite like monorepos to a certain size (eg: until they get big enough that the tooling we typically use outside of big tech starts to fall down). As an AI, I'm not sure that I care? I'd guess that context management can actually be easier if each microservice has a well documented API (openapi/graphql/grpc/asyncapi/whatever) and you provide the agent harness…

> The tedium of making branches / commits / pull requests across 6 repos to land a feature is less problematic to an agent.

You're probably not using your microservices correctly if you need to change more than one service at the same time. The whole point of microservices is independently developing and deploying the services. Sweeping changes like that should be done in pieces, one service at a time.

Which is why microservices are best for larger organizations, because it reduces coordination between dev groups.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#37

Post by ReaLNero »

wasmperson wrote: post #35

Alternatively: don't worry about keeping trunk "green" at all. Have a second branch called "stable" or something that auto-fast-forwards to latest trunk whenever trunk is green. Checkout stable, push new changes to trunk, avoid breaking CI, but if you break CI then don't worry about it, just push up a fix. If you act like trunk is this "sacred" thing that must always be ready to deploy then what you end up with is a…

I don’t understand how this would work.

If you’re using Git and you branch off ‘stable’, you won’t be able to merge to ‘trunk’ unless you rebase to pick up all of its changes. If a commit were to be reverted in the trunk now you need to revert it from all PRs as well. In the end, it’s equivalent to having one branch.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#38

Post by kyrra »

fragmede wrote: post #22
kyrra wrote: post #17

Earlier quoted context omitted.

The problem with many of these tools is they are built on lots of other tech, like S3 dev called out: https://x.com/haipingfu/status/2084880266858995990 Piper is built on lots of other tech like spanner and chubby and many others. And some of those have specific hardware requirements that only exist in Google datacenters. Untangling that web of tech is neigh impossible (or not something worth the cost to leadership).

Why is that a problem? They sell Spanner on GCP. Sell Piper as well! It's obvious as shit but Google doesn't know what it's doing anymore. They should have bought GitHub, not Microsoft, and then we wouldn't have had the problems with scaling GitHub either.

Piper only works with CitC. Which is effectively a virtual file system when mounted locally. And when you have a monorepo that big, standard tools that want to scan an entire directory will stop working. Every part of the developer tool chain has to change to deal with code base that big.

I agree that Google should have bought github. The tech they built to scale Google Code I think would have helped a ton with GitHub.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#39

Post by bob1029 »

I struggle with how these ideas meaningfully reduce the fundamental coordination problem.

The merge queue does provide serialization over change sets, but the ordering of these changes is very particular. Often, one change needs to go in before another change. The most common example I've experienced being database migrations.

It is a problem that ultimately must be solved externally. Speculatively deriving arbitrary order is likely to generate far more noise than signal in meaningfully complex merge scenarios. How many speculative orderings are we going to spend energy investigating before we find one we like / trust?

If you are in a situation where you have so much conflicting work that you think a specialized robot needs to be employed to sort it out, then you need to back the hell away from the technology and have a human conversation about the business and everyone's role in it.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#40

Post by conradludgate »

wasmperson wrote: post #35

Alternatively: don't worry about keeping trunk "green" at all. Have a second branch called "stable" or something that auto-fast-forwards to latest trunk whenever trunk is green. Checkout stable, push new changes to trunk, avoid breaking CI, but if you break CI then don't worry about it, just push up a fix. If you act like trunk is this "sacred" thing that must always be ready to deploy then what you end up with is a…

How does this work with a monorepo with hundreds of teams around the globe? If you break the trunk, that means no other teams can release
Post reply on HN