Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Orchestrating AI code review at scale

blog.cloudflare.com

Post reply on HN
Print view View original

51–60 of 66 posts

Re: Orchestrating AI code review at scale

#51

Post by MeetingsBrowser »

alain94040 wrote: post #34
OtherShrezzing wrote: post #32

>Code review is a fantastic mechanism for catching bugs and sharing knowledge "Sharing knowledge" is one of the first phrases in the article, and highlighted as a key benefit of code review. But the loss to human-capital from this process is never examined in the post. > Trivial reviews (typo fixes, small doc changes) cost 20 cents on average They did around 25,000 of these runs (about 20% of total). So CF spent $5k…

the labour cost of having an intern/entry level engineer spend ~30-60s looking through these is likely close to $0.20 Did you do the math? Your estimate feels way off. First, I doubt an intern would process one PR in 30s. Maybe 2-3 minutes, to read 10 lines carefully looking for typos and indentation mistakes. We pay interns close to $100K these days (in a company like CloudFlare), so that's ~80c/minute. My estimate…

$100k is probably at the top end. Even at Cloudflare, interns are more likely to be making $25-40/hour according to levels.fyi, which works out to $52-83k.

Re: Orchestrating AI code review at scale

#52

Post by esafak »

thih9 wrote: post #8

> Today, when an engineer at Cloudflare opens a merge request, it gets an initial pass from a coordinated smörgåsbord of AI agents. I’d prefer to have that happen as some sort of pre commit hook, before a merge request is sent. The feedback loop might be a bit faster and the process might produce less noise this way.

You can and should run your reviews beforehand but the same reviews should run in CI too (just like with commit hooks) because reviews are nondeterministic and for verification (even if they were deterministic).

Re: Orchestrating AI code review at scale

#53

Post by 34qa123 »

The suits have taken over Cloudflare. All buzzwords are on the bingo card: Using Bun, modeling agent roles after management, graphs, you name it.

They apparently think they need to cash in on AI by serving models and at the same time blocking scrapers. So they need to fuel the hype by pretending to use it.

This shows how the US economy is fundamentally broken: companies that provide a useful service (in theory, if you discount SSL MITM and turnstile gatekeeping) struggle, quasi-religious scams like OpenAI and Anthropic get funded by mentally ill Boomers and Gen-Xers.

Re: Orchestrating AI code review at scale

#54

Post by FuckButtons »

onlyrealcuzzo wrote: post #44
OtherShrezzing wrote: post #32

>Code review is a fantastic mechanism for catching bugs and sharing knowledge "Sharing knowledge" is one of the first phrases in the article, and highlighted as a key benefit of code review. But the loss to human-capital from this process is never examined in the post. > Trivial reviews (typo fixes, small doc changes) cost 20 cents on average They did around 25,000 of these runs (about 20% of total). So CF spent $5k…

Well, AI costs are definitely going to go down at least 90% in the next ~18 months for the same quality of output (and probably 90% again in the 24 months after). Are you sure it's going to make sense to pay someone to do that moving forward? I don't think it's worth it now, by the way. It's definitely not going to be worth it in the near future. Can we even blink for $0.002? What happens when the next 90% increase i…

> Well, AI costs are definitely going to go down at least 90% in the next ~18 months for the same quality of output (and probably 90% again in the 24 months after

As far as I can see, token costs have been steadily increasing over the past few months, so I’m not sure that buying the hype that another 90% cost reduction is just around the corner is warranted.

Re: Orchestrating AI code review at scale

#55

Post by rzmmm »

krzyk wrote: post #36
rzmmm wrote: post #2

> The entire system also runs locally. I think approaches like this don't need to run other than locally. Maybe integrated as pre-push hook. The system is nondeterministic, so it's at odds with the purpose of CI.

It is starting the review during CI (CI just triggers the review), not blocking merges like failed build or lint failures.

[dead]

Re: Orchestrating AI code review at scale

#56

Post by Zopieux »

onlyrealcuzzo wrote: post #44
OtherShrezzing wrote: post #32

>Code review is a fantastic mechanism for catching bugs and sharing knowledge "Sharing knowledge" is one of the first phrases in the article, and highlighted as a key benefit of code review. But the loss to human-capital from this process is never examined in the post. > Trivial reviews (typo fixes, small doc changes) cost 20 cents on average They did around 25,000 of these runs (about 20% of total). So CF spent $5k…

Well, AI costs are definitely going to go down at least 90% in the next ~18 months for the same quality of output (and probably 90% again in the 24 months after). Are you sure it's going to make sense to pay someone to do that moving forward? I don't think it's worth it now, by the way. It's definitely not going to be worth it in the near future. Can we even blink for $0.002? What happens when the next 90% increase i…

Is the price reduction in the room with us right now?

Re: Orchestrating AI code review at scale

#57

Post by [deleted] »

onlyrealcuzzo wrote: post #44
OtherShrezzing wrote: post #32

>Code review is a fantastic mechanism for catching bugs and sharing knowledge "Sharing knowledge" is one of the first phrases in the article, and highlighted as a key benefit of code review. But the loss to human-capital from this process is never examined in the post. > Trivial reviews (typo fixes, small doc changes) cost 20 cents on average They did around 25,000 of these runs (about 20% of total). So CF spent $5k…

Well, AI costs are definitely going to go down at least 90% in the next ~18 months for the same quality of output (and probably 90% again in the 24 months after). Are you sure it's going to make sense to pay someone to do that moving forward? I don't think it's worth it now, by the way. It's definitely not going to be worth it in the near future. Can we even blink for $0.002? What happens when the next 90% increase i…

[deleted]

Re: Orchestrating AI code review at scale

#58

Post by afro88 »

> When we first started experimenting with AI code review, we took the path that most other people probably take: we tried out a few different AI code review tools and found that a lot of these tools worked pretty well, and a lot of them even offered a good amount of customisation and configurability! Unfortunately, though, the one recurring theme that kept coming up was that they just didn’t offer enough flexibility and customisation for an organisation the size of Cloudflare.

Most people I know had the experience that signal to noise was way off, regardless of scale. So it was a burden rather than a help. Code review by AI ended up being a skill before creating the PR so the dev owning the PR addressed everything before the team got bogged down with it in review

Re: Orchestrating AI code review at scale

#59

Post by notawhitemale »

OtherShrezzing wrote: post #32

>Code review is a fantastic mechanism for catching bugs and sharing knowledge "Sharing knowledge" is one of the first phrases in the article, and highlighted as a key benefit of code review. But the loss to human-capital from this process is never examined in the post. > Trivial reviews (typo fixes, small doc changes) cost 20 cents on average They did around 25,000 of these runs (about 20% of total). So CF spent $5k…

[dead]
Post reply on HN