Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Orchestrating AI code review at scale

blog.cloudflare.com

Post reply on HN
Print view View original

21–30 of 66 posts

Re: Orchestrating AI code review at scale

#21

Post by nullbio »

Cthulhu_ wrote: post #20
thih9 wrote: post #8

> Today, when an engineer at Cloudflare opens a merge request, it gets an initial pass from a coordinated smörgåsbord of AI agents. I’d prefer to have that happen as some sort of pre commit hook, before a merge request is sent. The feedback loop might be a bit faster and the process might produce less noise this way.

I have mixed feelings, but it boils down to how long it takes and / or cost. Pre-commit hooks should be fast, as it's something you'd do (normally) a few dozen times a year. I don't believe sending a review job to a remote agent is fast, nor will waiting for a review to finish a commit be good for anyone. CI on the other hand can be slower and runs async, it's fire and forget so you can switch tasks. If noise is an i…

Few dozen times a year?

Re: Orchestrating AI code review at scale

#22

Post by throwaway613746 »

plmpsu wrote: post #4

I built a more naive version for our team using Copilot and GitHub actions and it works quite well (wish I had metrics too). The team loves it. The ROI here is so high that I don't mind using the strongest model available for the actual code review. I don't trust Sonnet and such. Just let Opus or GPT 5.5 do the whole thing and pay a bit more for less complexity.

[dead]

Re: Orchestrating AI code review at scale

#23

Post by trollbridge »

Cthulhu_ wrote: post #20
thih9 wrote: post #8

> Today, when an engineer at Cloudflare opens a merge request, it gets an initial pass from a coordinated smörgåsbord of AI agents. I’d prefer to have that happen as some sort of pre commit hook, before a merge request is sent. The feedback loop might be a bit faster and the process might produce less noise this way.

I have mixed feelings, but it boils down to how long it takes and / or cost. Pre-commit hooks should be fast, as it's something you'd do (normally) a few dozen times a year. I don't believe sending a review job to a remote agent is fast, nor will waiting for a review to finish a commit be good for anyone. CI on the other hand can be slower and runs async, it's fire and forget so you can switch tasks. If noise is an i…

You must have some pretty monster commits.

Re: Orchestrating AI code review at scale

#24

Post by cush »

rhgraysonii wrote: post #10
thih9 wrote: post #8

> Today, when an engineer at Cloudflare opens a merge request, it gets an initial pass from a coordinated smörgåsbord of AI agents. I’d prefer to have that happen as some sort of pre commit hook, before a merge request is sent. The feedback loop might be a bit faster and the process might produce less noise this way.

Valid, but you lose the lived history that comes with the audit log of it being actual review back and forth and CI runs vs lost to a developers machine and only a relic in the commit log. I can see both sides, though.

People usually squash merge anyways

Re: Orchestrating AI code review at scale

#25

Post by criley2 »

faangguyindia wrote: post #16

what's best workflow for solo devs?

You can do basically the same thing as cloudflare except as a skill you run in your local harness. If you're going through the motions with PRs and are familiar with actions, you can have it run in a github action instead. But this is basically just a skill. The Claude code review skill is a simple version of exactly this.

Re: Orchestrating AI code review at scale

#26

Post by suika »

As a solo dev or rather nowadays more so only a decision maker / agent overseer, I came to enjoy letting my agents develop against a Gerrit repository / workflow. Dev agent pushes a CL, review agent picks it up (not just the diff, but the full repo), runs tests/reviews/review-subagents and concludes by posting a review as well as a vote. This goes back and forth with new patch sets / replies to the threads. Eventually the CL gets a +2 or whatever and I have the final call to manually submit it. It is way slower compared to just pushing through development with one agent doing everything yolo against a normal repository, but it seems to me that the additional time is well spent (no, I don't have fancy graphs or similar analysis to prove this other than my gut feeling after looking at recent development results).

Re: Orchestrating AI code review at scale

#27

Post by joshuamoyers »

we’ve been struggling with review throughput. this actually seems worthwhile to build at this point though i remain fairly skeptical of workflows that are agent-only, at a point it seems like the only practical solution.

we are finding lots of value in self review. its the “imagine you are doing a synchronous paired review with someone - anything that is difficult to explain, has a code smell, doesnt fit the architecture of the system around you, write a comment.” then at the end, agents do a good job of looping over PR comments.

the second thing would be a guided, educational code review tool - there are a few attempts at this, but nothing that has a good enough interface to actually stick. organize hunks by semantic importance, spend some tokens exploring the surrounding systems, showing how new code, public apis and data model flow with the existing design, and allow a human to traverse larger PRs more quickly.

thank you to cloudflare for publishing this.

Re: Orchestrating AI code review at scale

#28

Post by SpicyLemonZest »

Zanfa wrote: post #17
rhgraysonii wrote: post #10

Earlier quoted context omitted.

Valid, but you lose the lived history that comes with the audit log of it being actual review back and forth and CI runs vs lost to a developers machine and only a relic in the commit log. I can see both sides, though.

Can you elaborate about the practical value of having the history of back and forth, in a PR or even in the commit log? In my 20ish years of experience, I can’t recall a single instance where I’ve solved something thanks to having this work-in-progress state persisted in the repo history. It’s exclusively been the other way around where having a smaller number of larger squished commits (post merge) that’s made the p…

It's not about having it in the commit history. I've seen a few cases where the back and forth revealed that the AI reviewer was offering bad advice (and a few others where I suspect bad local AI advice is why people keep sending me the same category of mistake).

Re: Orchestrating AI code review at scale

#29

Post by NiloCK »

thih9 wrote: post #8

> Today, when an engineer at Cloudflare opens a merge request, it gets an initial pass from a coordinated smörgåsbord of AI agents. I’d prefer to have that happen as some sort of pre commit hook, before a merge request is sent. The feedback loop might be a bit faster and the process might produce less noise this way.

Like it or not, the "merge request" (eg, open a PR) is the Schelling point of relevant information. I expect that At scale here refers to size of software projects, and not only code velocity. Software projects of large enough size have CI configuration that don't typically fully-run on each dev machine.

Re: Orchestrating AI code review at scale

#30

Post by mock-possum »

thih9 wrote: post #8

> Today, when an engineer at Cloudflare opens a merge request, it gets an initial pass from a coordinated smörgåsbord of AI agents. I’d prefer to have that happen as some sort of pre commit hook, before a merge request is sent. The feedback loop might be a bit faster and the process might produce less noise this way.

Pre commit only happens on your machine though - you lose the ability to have a shared review surface where you can tag others on your team to specifically prompt discussion or verification on issues that touch their domain. When an agent points out a potential security issue with how my work ties into infrastructure, I want to just be able to tag our infra team and ask “hey is this something to worry about?” The agent, myself, and the other team member have now all contributed to a threaded discussion that is easily referencable in the future.
Post reply on HN