Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Uber SubmitQueue: a high-performance speculative merge queue

github.com

Post reply on HN
Print view View original

51–60 of 63 posts

Re: Uber SubmitQueue: a high-performance speculative merge queue

#51

Post by mac-mc »

IshKebab wrote: post #41
sdfhbdf wrote: post #27

I'm reading from the repo and I'm a little dumbfounded what's the innovation. > SubmitQueue speculatively rebases and validates multiple changes in parallel against predicted future states of HEAD. When validations pass, changes land automatically. When they fail, SubmitQueue isolates the offending change and retries the rest — all without human intervention. This seems to be a feature of GitHub (We have it in an old…

I think it's much cleverer about the order in which it tries to merge PRs, and how they are grouped. GitHub merge queues just do it in the order you click. Also I'm pretty sure it predates GitHub's implementation by a fair bit. Uber were one of the first people to do this. Maybe the first IIRC.

Yes this existed at least since 2018 or 2019 at Uber.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#52

Post by mac-mc »

jedberg wrote: post #18

I believe that the solution to the coordination problem is good monorepo tooling (like the OP) plus AI to understand the whole codebase and help the engineers understand how their part fits in. I was one of the biggest proponents of microservices, going so far as to traveling around the world spreading the gospel of microservices keynoting large tech conferences. I believed that microservices were the best solution t…

IMO AI agents harnesses (claude code, codex, etc) haven't added monorepo features yet as of a couple of months ago and thats what makes them painful. Basic things like, only apply these skills to the subdir that the .agents/skills directory exists in would go a long way. Or even reading the skills in a subdir .agents/skills directory. Or the ability to specify the basic monorepo custom VCS and other actions in a way that isn't limited to fragile skills and AGENTS.md specs that can get forgotten or unused as the context windows grows in a session and so on.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#53

Post by mac-mc »

jedberg wrote: post #50
claytonjy wrote: post #46

Earlier quoted context omitted.

if service A calls service B, and service B adds a new endpoint, or a new optional argument, service A needs an update to take advantage of it if a library is used by multiple services, and gets an important bug fix, each service using the library needs to update to get the fix these are sequences of changes, not literally at the same time or requiring deployment coordination, but when this happens a lot people start…

But a monorepo wouldn't make any of that easier. You have to make the change to service A and B, and then test both and deploy both. You haven't saved any time or effort in a monorepo. A library needs updating, you still have to update it and then test and deploy every service that relies on it independently. Again you haven't saved any time or effort. In fact you've made it worse, because if those services are maint…

The big thing about monorepos in this case is making a change atomic across a set of projects. Multirepos can't do this 99% of the time and often add a lot more procedure to something close. It's inherently async.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#54

Post by egwor »

sdfhbdf wrote: post #27

I'm reading from the repo and I'm a little dumbfounded what's the innovation. > SubmitQueue speculatively rebases and validates multiple changes in parallel against predicted future states of HEAD. When validations pass, changes land automatically. When they fail, SubmitQueue isolates the offending change and retries the rest — all without human intervention. This seems to be a feature of GitHub (We have it in an old…

I've been doing this for years (8+ at most recent place?) and we did it about 15years ago at place before. We keep having to relearn these features, which is on eof the reasons why some groups are far more effective than others

Re: Uber SubmitQueue: a high-performance speculative merge queue

#55

Post by claytonjy »

jedberg wrote: post #50
claytonjy wrote: post #46

Earlier quoted context omitted.

if service A calls service B, and service B adds a new endpoint, or a new optional argument, service A needs an update to take advantage of it if a library is used by multiple services, and gets an important bug fix, each service using the library needs to update to get the fix these are sequences of changes, not literally at the same time or requiring deployment coordination, but when this happens a lot people start…

But a monorepo wouldn't make any of that easier. You have to make the change to service A and B, and then test both and deploy both. You haven't saved any time or effort in a monorepo. A library needs updating, you still have to update it and then test and deploy every service that relies on it independently. Again you haven't saved any time or effort. In fact you've made it worse, because if those services are maint…

I haven’t worked at a Big Tech monorepo place, but I hope/assume that in the service dependency case, you would touch both in one changeset and the deploy tooling understands the dependency tree and orders the deployments correctly. Without that i agree its the polyrepo approach just with all PRs in the same repo instead of multiple.

In the library scenario, I know tooling like bazel ensures that’s one changeset, not 3+. Tests run for both the changed library and all consumers of it in the same pass. You’re right that it might loop in others for review who weren’t expecting it, but i think that’s the same in a polyrepo approach.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#56

Post by jedberg »

mac-mc wrote: post #53
jedberg wrote: post #50

Earlier quoted context omitted.

But a monorepo wouldn't make any of that easier. You have to make the change to service A and B, and then test both and deploy both. You haven't saved any time or effort in a monorepo. A library needs updating, you still have to update it and then test and deploy every service that relies on it independently. Again you haven't saved any time or effort. In fact you've made it worse, because if those services are maint…

The big thing about monorepos in this case is making a change atomic across a set of projects. Multirepos can't do this 99% of the time and often add a lot more procedure to something close. It's inherently async.

> The big thing about monorepos in this case is making a change atomic across a set of projects.

The thing microservices is supposed to solve is eliminating the need to make atomic changes across services. There is an inherent overhead in doing that, like making your service support both new and old APIs as long as any service exists using the old API. This is the cost of microservices.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#57

Post by handfuloflight »

plumeria wrote: post #42
fcarraldo wrote: post #31

Earlier quoted context omitted.

I work at a monorepo company and any time someone gets to work on a project that necessitates working outside of the monorepo, it's a night-and-day improvement. Tools, especially open source ones (linters, static analysis, scanning, LSPs, IDEs, etc) are not built for monorepos, and with AI Agents working in the monorepo results in an enormous increase in input tokens as the agents are constantly trying to grep this g…

> Tools, especially open source ones (linters, static analysis, scanning, LSPs, IDEs, etc) are not built for monorepos, and with AI Agents working in the monorepo results in an enormous increase in input tokens as the agents are constantly trying to grep this giant source tree. I've been thinking that one could dynamically patch .claude/settings.json (or its equivalent for other agents) to allow reads/writes only to…

This is a great idea.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#58

Post by malfist »

fastball wrote: post #29
twic wrote: post #26

Earlier quoted context omitted.

> A well done monorepo is a huge force multiplier on a large organization How so? I work at a company which uses a monorepo, and I haven't seen any upside to it yet. We have a tools team that's invested a vast amount of work in it. Still seems strictly worse than a 'normal' polyrepo setup. I haven't understood why so many people are so enthusiastic about it.

Do you do any cross workspace/repo work, or are you mostly constrained to a single namespace?

Not op but I do cross repo work, but it's rightfully separate PRs as its separate services, separate contracts and separate deployments. Pretending a monorepo cross service PRs are contiguous is a recipe for deployment race conditions

Re: Uber SubmitQueue: a high-performance speculative merge queue

#59

Post by wasmperson »

ReaLNero wrote: post #37
wasmperson wrote: post #35

Alternatively: don't worry about keeping trunk "green" at all. Have a second branch called "stable" or something that auto-fast-forwards to latest trunk whenever trunk is green. Checkout stable, push new changes to trunk, avoid breaking CI, but if you break CI then don't worry about it, just push up a fix. If you act like trunk is this "sacred" thing that must always be ready to deploy then what you end up with is a…

I don’t understand how this would work. If you’re using Git and you branch off ‘stable’, you won’t be able to merge to ‘trunk’ unless you rebase to pick up all of its changes. If a commit were to be reverted in the trunk now you need to revert it from all PRs as well. In the end, it’s equivalent to having one branch.

> I don’t understand how this would work.

It's not all that strange: it's basically how version control worked before everybody switched to git. Anybody still on SVN is basically using this exact development model.

> If you’re using Git and you branch off ‘stable’, you won’t be able to merge to ‘trunk’ unless you rebase to pick up all of its changes.

Yeah, so you just do that. To be clear: most people would be doing their work on trunk. When I say "checkout stable" I mean "checkout stable if you need to start from something that passes CI". In the case where trunk passes CI or was recently passing CI this is basically the same as checking out trunk.

> If a commit were to be reverted in the trunk now you need to revert it from all PRs as well.

Most people push straight into trunk, no PRs needed. For the cases where you need PRs, reverting something in trunk won't affect them: `git revert` adds a new commit just like any other change.

If by "revert" what you actually meant was rolling back trunk to an earlier commit: don't do that.

Re: Uber SubmitQueue: a high-performance speculative merge queue

#60

Post by wasmperson »

conradludgate wrote: post #40
wasmperson wrote: post #35

Alternatively: don't worry about keeping trunk "green" at all. Have a second branch called "stable" or something that auto-fast-forwards to latest trunk whenever trunk is green. Checkout stable, push new changes to trunk, avoid breaking CI, but if you break CI then don't worry about it, just push up a fix. If you act like trunk is this "sacred" thing that must always be ready to deploy then what you end up with is a…

How does this work with a monorepo with hundreds of teams around the globe? If you break the trunk, that means no other teams can release

> How does this work with a monorepo with hundreds of teams around the globe? If you break the trunk, that means no other teams can release

If you break trunk you just fix it. If a release needs to happen urgently and trunk is broken then you can use stable, which is the last working version of trunk.

EDIT: RE: hundreds of teams: If it's a bunch of independently-built things (libraries, programs, or servers) that happen to share a repo then it will work fine. If you're talking a single mega-project like Linux then maybe not, but Linux is pretty exceptional.

Post reply on HN