Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Orchestrating AI code review at scale

blog.cloudflare.com

Post reply on HN
Print view View original

31–40 of 66 posts

Re: Orchestrating AI code review at scale

#31

Post by ramoz »

joshuamoyers wrote: post #27

we’ve been struggling with review throughput. this actually seems worthwhile to build at this point though i remain fairly skeptical of workflows that are agent-only, at a point it seems like the only practical solution. we are finding lots of value in self review. its the “imagine you are doing a synchronous paired review with someone - anything that is difficult to explain, has a code smell, doesnt fit the architec…

I do think Cloudflare probably institutes a similar manual review process as well. I have a handful of fairly vocal and supportive engineers I stay in contact with around https://plannotator.ai (there is an integrated code review surface that creates a feedback loop with your local agent).

> agents do a good job of looping over PR comments

This is the easy part. Most harnesses enable some sort of integration now, so you can actually create a smooth local experience around this as well - better code before it ships to more costly review or bloats PR threads.

> guided, educational code review tool

This is a bit tougher, and I find the main harness chat tends to work best. I learn better when I'm more engaged and aware of what I'm asking. It's easy to stick a code tour type of thing on a screen. It's hard to really nail the right attention and learning mechanism around it.

Re: Orchestrating AI code review at scale

#32

Post by OtherShrezzing »

>Code review is a fantastic mechanism for catching bugs and sharing knowledge

"Sharing knowledge" is one of the first phrases in the article, and highlighted as a key benefit of code review. But the loss to human-capital from this process is never examined in the post.

> Trivial reviews (typo fixes, small doc changes) cost 20 cents on average

They did around 25,000 of these runs (about 20% of total). So CF spent $5k in the period making language models run through PRs which were <10 lines long. I get that CF engineers are paid well, but the labour cost of having an intern/entry level engineer spend ~30-60s looking through these is likely close to $0.20, and that engineer builds some human-capital while they're at it.

Re: Orchestrating AI code review at scale

#33

Post by krzyk »

thih9 wrote: post #8

> Today, when an engineer at Cloudflare opens a merge request, it gets an initial pass from a coordinated smörgåsbord of AI agents. I’d prefer to have that happen as some sort of pre commit hook, before a merge request is sent. The feedback loop might be a bit faster and the process might produce less noise this way.

It is easier to view code review results in a tool and not in a text during commit.

There is no universal standard which IDEs support for code review results (there is SARIF, but it is not supported that widely). A review result on a web page with comments from humans, is valuable.

Re: Orchestrating AI code review at scale

#34

Post by alain94040 »

OtherShrezzing wrote: post #32

>Code review is a fantastic mechanism for catching bugs and sharing knowledge "Sharing knowledge" is one of the first phrases in the article, and highlighted as a key benefit of code review. But the loss to human-capital from this process is never examined in the post. > Trivial reviews (typo fixes, small doc changes) cost 20 cents on average They did around 25,000 of these runs (about 20% of total). So CF spent $5k…

the labour cost of having an intern/entry level engineer spend ~30-60s looking through these is likely close to $0.20

Did you do the math? Your estimate feels way off. First, I doubt an intern would process one PR in 30s. Maybe 2-3 minutes, to read 10 lines carefully looking for typos and indentation mistakes. We pay interns close to $100K these days (in a company like CloudFlare), so that's ~80c/minute. My estimate is therefore closer to $1.6 per PR. About 10X.

You are correct that there is a residual value with the intern, over time they would start learning (a little bit) about the code base.

Re: Orchestrating AI code review at scale

#35

Post by krzyk »

cush wrote: post #24
rhgraysonii wrote: post #10

Earlier quoted context omitted.

Valid, but you lose the lived history that comes with the audit log of it being actual review back and forth and CI runs vs lost to a developers machine and only a relic in the commit log. I can see both sides, though.

People usually squash merge anyways

Actually not, it is similar debate like rebase or merge.

e.g. I don't squash, I prefer to see full history, not redacted one.

Re: Orchestrating AI code review at scale

#36

Post by krzyk »

rzmmm wrote: post #2

> The entire system also runs locally. I think approaches like this don't need to run other than locally. Maybe integrated as pre-push hook. The system is nondeterministic, so it's at odds with the purpose of CI.

It is starting the review during CI (CI just triggers the review), not blocking merges like failed build or lint failures.

Re: Orchestrating AI code review at scale

#37

Post by krzyk »

plmpsu wrote: post #4

I built a more naive version for our team using Copilot and GitHub actions and it works quite well (wish I had metrics too). The team loves it. The ROI here is so high that I don't mind using the strongest model available for the actual code review. I don't trust Sonnet and such. Just let Opus or GPT 5.5 do the whole thing and pay a bit more for less complexity.

I did similarly with copilot.

I have about 15 or so subagents doing reviews from different perspectives (or providing some additional value, like finding agents.md files, doing confidence ranking, describing images attached to the PR, that get validated later on with Jira issue description).

I used it since about November, with large scale popularity in my company reaching in April - all that on a 300 premium requests (because they allowed starting subagents, and there was no limit how long a single request can last) - so it would cost something like $5000 and $8000 for April and May if it was API pricing. I had similar cost per review (about $0.90) with Opus 4.6 and help from Sonnet and Haiku for simpler tasks. It did about 4000 reviews during the last 2 months.

And starting in June, it will be dead because it will be API pricing and for $30 (or $19 since September) it will do just few reviews.

A fun project.

Re: Orchestrating AI code review at scale

#38

Post by Dumbledumb »

This blog post is full of small inconsistencies that make it read like a low quality SEO piece.

> We also extract a shared context file (shared-mr-context.txt) from the coordinator's prompt and write it to disk. Sub-reviewers read this file instead of having the full MR context duplicated in each of their prompts. This was a deliberate decision, as duplicating even a moderately-sized MR context across seven concurrent reviewers would multiply our token costs by 7x.

No, it would not, because neither is the prompt of the subagent 100% of its token usage, nor will the "shared-mr-context.txt" which is then being read have a size of zero compared to the creation of this shared context.

> You don't need seven concurrent AI agents burning Opus-tier tokens to review a one-line typo fix in a README.

Yeah, well you wouldn't have anyways. Earlier in the post it says that Opus is "exclusively for the Review Coordinator".

Re: Orchestrating AI code review at scale

#39

Post by Dumbledumb »

Dumbledumb wrote: post #38

This blog post is full of small inconsistencies that make it read like a low quality SEO piece. > We also extract a shared context file (shared-mr-context.txt) from the coordinator's prompt and write it to disk. Sub-reviewers read this file instead of having the full MR context duplicated in each of their prompts. This was a deliberate decision, as duplicating even a moderately-sized MR context across seven concurren…

Also this seems plain wrong. Input token caching has now idea whether you @include the file or copy the contents into the prompt. That is handled entirely by opencode and, all else being equal, has no bearing on the cache ability of a trace.

> Our cache hit rate sits at 85.7%, which saves us an estimated five figures compared to what we would pay at full input token pricing. This is partially thanks to the shared context file optimisation — sub-reviewers reading from a cached context file rather than each getting their own copy of the MR metadata, but also by using the exact same base prompts across all runs, across all merge requests.

Re: Orchestrating AI code review at scale

#40

Post by pramodbiligiri »

faangguyindia wrote: post #16

what's best workflow for solo devs?

Based on the section about "Specialised agents" (https://blog.cloudflare.com/ai-code-review/#specialised-agen...), I'd say create a bunch of review prompts and run it against the code? The rest of the blog post seems to be the engineering around it: for scale, cost, team size etc.
Post reply on HN