Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How are pull requests integrated in a repo with high commit frequency?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 71 posts

Re: Ask HN: How are pull requests integrated in a repo with high commit frequency?

#41

Post by lhorie »

The technology you're looking for goes by many names, "submit queue", "merge queue", "merge train", etc.

In a nutshell, the idea is to keep track of currently running jobs, then any time a new commit enters the queue, you merge all the running commits into that and test that as a bundle. If merging fails, bail out. If that bundle job fails, bail out. If one of the previously running commits fail, you bail out of the bundle job and run another speculative bundle job without the failed commit. When a bundle succeeds, land all of its commits and abort any remaining redundant jobs.

Such systems will sometimes have heuristics to bundle commits together in some smarter way than a naive queue (e.g. preferring to bundle of commits without overlapping changes, or not bundling small commits with huge ones to prevent infecting speculative builds with the slowness of the big commit, using AI to come up with heuristics to detect "likely to fail" commits and preemptively starting speculative builds without them, etc)

Re: Ask HN: How are pull requests integrated in a repo with high commit frequency?

#42

Post by jFriedensreich »

we rebase all branches on top of the target branch and only allow fast forward merges, which prevents any untested merge commits to cause issues and is really good for reading history. if there are changes to the target branch since the last rebase, there needs to be another rebase triggering automatic CI/CD. if there is a known bottleneck where this would happen a lot to a number of PRs and all the prs are green we rebase the prs on each other in the desired merge order just have to wait for the youngest PR to run through ci/cd and then can fast forward merge all at once.

Re: Ask HN: How are pull requests integrated in a repo with high commit frequency?

#43

Post by pxeger1 »

mijoharas wrote: post #23
apetresc wrote: post #5

Check out something like Bors, which implements a paradigm that fixes this class of issue: https://github.com/bors-ng/bors-ng (Homepage: https://bors.tech/ )

Does bors stand for something? what does the ng stand for? (I wasn't able to find out while googling.)

-ng is a suffix sometimes used for "reboots" of abandoned or poorly maintained software; I believe it stands for "next generation".

Re: Ask HN: How are pull requests integrated in a repo with high commit frequency?

#44

Post by henryfjordan »

rrss wrote: post #29
apetresc wrote: post #5

Check out something like Bors, which implements a paradigm that fixes this class of issue: https://github.com/bors-ng/bors-ng (Homepage: https://bors.tech/ )

does bors limit the commit frequency / PR merge frequency to one merge every test runtime (5 minutes in OP), or does it do something more complex and speculate / run some stuff in parallel? OP’s numbers are 120 commits per hour and 5 minute test run, I’m wondering if adopting bors would necessarily reduce that to 12 commits per hour.

Bors batches PRs, tests the batch, then merges the whole batch at once. It's made to handle high-throughput repos.

Re: Ask HN: How are pull requests integrated in a repo with high commit frequency?

#45

Post by krab »

rrss wrote: post #29
apetresc wrote: post #5

Check out something like Bors, which implements a paradigm that fixes this class of issue: https://github.com/bors-ng/bors-ng (Homepage: https://bors.tech/ )

does bors limit the commit frequency / PR merge frequency to one merge every test runtime (5 minutes in OP), or does it do something more complex and speculate / run some stuff in parallel? OP’s numbers are 120 commits per hour and 5 minute test run, I’m wondering if adopting bors would necessarily reduce that to 12 commits per hour.

It tries to build all queued changes together at first. So if you don't have any conflicts, there is no throughput limit.

Re: Ask HN: How are pull requests integrated in a repo with high commit frequency?

#46

Post by geraldcombs »

heavenlyblue wrote: post #18
chrisshroba wrote: post #10

Earlier quoted context omitted.

This is a great solution and while I was at Google I noticed several high commit frequency teams using this strategy. Of course Google had built a bunch of custom tooling and infrastructure around it, so I can’t vouch for how easy it would be to integrate into a different company’s dev workflow, but if there are enough developers to make it worthwhile, then tasking a few developers with setting this up should be a us…

GitLab has this feature.

Not for fast forward merges.

Re: Ask HN: How are pull requests integrated in a repo with high commit frequency?

#47

Post by msk20 »

There is ZUUL Gating[0] CI it is actually the perfect solution for this. It works with Github or Git based repository system.

It automatically tests the changes with a simulated merge on master together. So it orders PR1 -> PR2 -> PR3 -> .... -> PR-100 by order of approval. If PR1 -> PR2 (Fails) -> PR3 -> .... -> PR-100

It restarts -> PR3 -> .... -> PR-100 and Up after removing PR2. This behavior is even customizable.

Video of it in action: https://zuul-ci.org/media/simulation.webm

Links: [0]:https://zuul-ci.org/

Re: Ask HN: How are pull requests integrated in a repo with high commit frequency?

#49

Post by lhorie »

jd__ wrote: post #32
jakub_g wrote: post #22

This is a real problem in my current place (huge monorepo, 100+ devs, 100+ merges per day). Like others said, "merge queue" is the solution. GitHub's one has been in beta for many months now. There are also dedicated companies doing this like https://mergify.com/ But there's several tricky aspects like "what if I want some commit to jump in front of queue?", compliance etc. Due to the multitude of requirements and of…

I'd be curious to know what you find limited in off-the-shelf solutions? (disclaimer: I'm from the Mergify team )

One question and one comment.

Question: You list Uber (my employer) as a user in your homepage, but searching internally, the only search result for mergify is a mergify.yml in some random debian module for lua, so it doesn't seem like we actually use it for anything (and we built an in-house solution for the repos w/ serious commit traffic). I'm curious how you determine who your corporate users are?

Comment: we ended up building a merge queue in house because frankly build/test pipelines can look very different,(e.g. at one point we had jenkins pipelines and buildkite pipelines, some using bazel, some not, talking to phabricator instead of github, etc). At the end of day plugging in a merge queue technology requires a bunch of work integrating with a myriad of things anyways. Since the engineering effort is relatively high regardless, doing it in house lets us experiment with more aggressive optimization heuristics than just waiting on a 3rd party.

Re: Ask HN: How are pull requests integrated in a repo with high commit frequency?

#50

Post by yakubin »

At dayjob we have a "commit queue" server. When a patch is accepted on Gerrit, there is a button to send it to the queue. The queue cherry-picks the commit onto the current master branch, makes several builds (different configs) and runs tests. If everything is ok, it pushes the commit to master and marks the patch on Gerrit as merged. If any of the builds or tests fail, it doesn't push the change and instead makes a comment on Gerrit with a link to the broken build/test.

So all commits are made sequentially, but most of the time developers don't need to rebase them themselves before pushing. The time distribution of commits sent to the queue is non-uniform, so you may have a long queue during the day, but by late evening it's pretty much empty. It has the greatest number of commits on Fridays.

Post reply on HN