Orchestrating AI code review at scale
blog.cloudflare.com
Orchestrating AI code review at scale
1–10 of 66 posts
Re: Orchestrating AI code review at scale
#2> The entire system also runs locally.
I think approaches like this don't need to run other than locally. Maybe integrated as pre-push hook. The system is nondeterministic, so it's at odds with the purpose of CI.
Re: Orchestrating AI code review at scale
#3> The entire system also runs locally. I think approaches like this don't need to run other than locally. Maybe integrated as pre-push hook. The system is nondeterministic, so it's at odds with the purpose of CI.
[dead]
Re: Orchestrating AI code review at scale
#4I built a more naive version for our team using Copilot and GitHub actions and it works quite well (wish I had metrics too). The team loves it.
The ROI here is so high that I don't mind using the strongest model available for the actual code review. I don't trust Sonnet and such. Just let Opus or GPT 5.5 do the whole thing and pay a bit more for less complexity.
Re: Orchestrating AI code review at scale
#5> The entire system also runs locally. I think approaches like this don't need to run other than locally. Maybe integrated as pre-push hook. The system is nondeterministic, so it's at odds with the purpose of CI.
I'd argue it's pretty much like monitoring, which certainly benefits from multiple people seeing the same stats and alerts. I agree it's at odds with CI/CD and should probably not block anything, like deterministic checks commonly do.
Re: Orchestrating AI code review at scale
#6> The entire system also runs locally. I think approaches like this don't need to run other than locally. Maybe integrated as pre-push hook. The system is nondeterministic, so it's at odds with the purpose of CI.
I'm not sure the people integrating it into CI process understand what CI is.
Re: Orchestrating AI code review at scale
#7> One of the operational headaches we didn’t predict was that large, advanced models like Claude Opus 4.7 or GPT-5.4 can sometimes spend quite a while thinking through a problem, and to our users this can make it look exactly like a hung job.
I had the same problem in my recursive agent harness. It would always come back, but it could sometimes take up to 10 minutes depending. I fixed this by adding a required "purpose" argument to every tool and call/return event. As the recursive evaluation proceeds, every single thing that happens streams incremental purpose text to the user's browser (also using the magic of JSONL for this). The incremental progress events contain the purpose and a detail section (tool arg JSON) that the user can expand/collapse.
Re: Orchestrating AI code review at scale
#8> Today, when an engineer at Cloudflare opens a merge request, it gets an initial pass from a coordinated smörgåsbord of AI agents.
I’d prefer to have that happen as some sort of pre commit hook, before a merge request is sent. The feedback loop might be a bit faster and the process might produce less noise this way.
Re: Orchestrating AI code review at scale
#9I built a more naive version for our team using Copilot and GitHub actions and it works quite well (wish I had metrics too). The team loves it. The ROI here is so high that I don't mind using the strongest model available for the actual code review. I don't trust Sonnet and such. Just let Opus or GPT 5.5 do the whole thing and pay a bit more for less complexity.
do you also have separate prompts for each domain (security, architecture etc?).
would love to look into it if any part of it is open source
Re: Orchestrating AI code review at scale
#10> Today, when an engineer at Cloudflare opens a merge request, it gets an initial pass from a coordinated smörgåsbord of AI agents. I’d prefer to have that happen as some sort of pre commit hook, before a merge request is sent. The feedback loop might be a bit faster and the process might produce less noise this way.
Valid, but you lose the lived history that comes with the audit log of it being actual review back and forth and CI runs vs lost to a developers machine and only a relic in the commit log. I can see both sides, though.