Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-6 Astra in code review: Gains, privacy, and cost

coderabbit.ai

Post reply on HN
Print view View original

61–70 of 77 posts

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#61

Post by vb-8448 »

tosh wrote: post #45

I ran a few toy benches comparing Astra with Sol and found Astra ~30% faster and at similar cost to Sol for the same outcome https://x.com/__tosh/status/2096201900555170032 the token efficiency helps Astra even though sticker price is 2.5x that of Sol

Same experience here, but I have some strange feeling.

Sol I'm used to working a month ago doesn't feel the same I'm using today, slower and less accurate. My gut feeling is that they quantize previous models to prioritize new ones and, who nows, make the new one look better.

Up to July I was using mostly anthropic models and the feeling was the same, so much so that I was able to predict every model release 1 or 2 days before public announcements.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#62

Post by pipes »

CuriouslyC wrote: post #55
ZephyrBlu wrote: post #43

Earlier quoted context omitted.

Thinking of AI-generated code and AI code reviews as the same "AI" is not correct. The reviewer is using a fresh context window with no previous knowledge of the changes. That is why it's powerful, because you get the agent to interrogate the code without any preconception about the changes. I found the Devin reviewer to be very good, and have heard good things about Cursor's Bugbot. I've also found asking an agent w…

Specialized AI code review software is so pointless though. Back when agents were dumb about git surgery and tool use it might have had a purpose, but now you could replace coderabbit with a skill and I bet the results would be better in some cases.

Mechanical code review by something like sonar qube is much cheaper. Use that for low bar quality gate and AI after.

AI code review is startlingly effective. Continually finding things me and my colleagues never would. Well the decent models do. Maybe not so much the cheap ones.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#63

Post by stingraycharles »

ZephyrBlu wrote: post #43
stingraycharles wrote: post #9

Earlier quoted context omitted.

Yeah I personally don’t understand the point of AI code review tools all that much, as AI is already generating the code as well. All of these AI code review tools create so much noise, yet don’t catch the really important things.

Thinking of AI-generated code and AI code reviews as the same "AI" is not correct. The reviewer is using a fresh context window with no previous knowledge of the changes. That is why it's powerful, because you get the agent to interrogate the code without any preconception about the changes. I found the Devin reviewer to be very good, and have heard good things about Cursor's Bugbot. I've also found asking an agent w…

I’m not saying that it’s not useful, I’m saying that it’s not useful in a “human in the loop” situation. This type of AI-to-AI review should be done agent-to-agent, not through Github PRs with tools like Devin.

In a manual review, I then expect all “machinery” to already be properly reviewed, and can focus on design / architecture. I would like an AI assisted review tool to make that part easier, not do the actual review for me.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#64

Post by ptrl600 »

stingraycharles wrote: post #9
eyalitki wrote: post #7

Comparison was done in the scope of coderabbit AI code review tool, which sadly makes it practically irrelevant. My personal experience as a software engineer, and a former security researcher who did manual code audit, is that this code review tool has such poor results that it isn't worth the "noise" and friction it causes developers during C/I code review

Yeah I personally don’t understand the point of AI code review tools all that much, as AI is already generating the code as well. All of these AI code review tools create so much noise, yet don’t catch the really important things.

They're pretty good for me, because I am still writing all the code, and it tends to catch the sorts of things humans mess up.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#66

Post by sdeframond »

mobelkh wrote: post #30
sdeframond wrote: post #28

How do you guys review AI-generated code ? In our team, frontend work is vibe-coded by the PO and merged as-is without review. Backend is coded by developers, using AI but in a slower, more controlled way. Recently, our PO has been trying his hand at vibe-coding the backend. I must say he is a smart guy, almost technical but not quite a developer. We've just been handed a burst of stacked PRs amounting for ~15k LOC b…

throw his garbage out, the time and effort taken to review that is magnitudes more than what it took to prompt it. have him start with an overall design doc if his change is 15k, it's definitely worth a design doc. and then have his contributions reviewed in pieces of 200-300 LoC PRs. any other solution is trading stability and system knowledge, that's 15k LoC no one is truly familiar with, even if you do try to revi…

He's made a bunch of 1-2k LOC PRs and there is a design doc. Everything is AI generated.

The issue is, if he generates all of that without reviewing the code, he will always be far faster than us. And he can't review the code. No matter how he slices it.

Also, he is the CPO/CTO. So we can say no, but there is a natural incentive to go his way. He still doesn't feel confident enough to just bypass the programmers and he's probably right. But it'd nice to find a way to use my expertise to review this amount of code meaningfully, somehow.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#67

Post by sdeframond »

fabianlindfors wrote: post #49
eterps wrote: post #48

Earlier quoted context omitted.

Which VM technology do you use?

Firecracker, with some tiny modifications to better manage memory for the deep nesting of forks

Would you mind sharing your infra budget needed to spin these VMs ?

Surely it is reasonable, but also way more than our budget. Id like to compare.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#68

Post by sdeframond »

embedding-shape wrote: post #39
mobelkh wrote: post #30

Earlier quoted context omitted.

throw his garbage out, the time and effort taken to review that is magnitudes more than what it took to prompt it. have him start with an overall design doc if his change is 15k, it's definitely worth a design doc. and then have his contributions reviewed in pieces of 200-300 LoC PRs. any other solution is trading stability and system knowledge, that's 15k LoC no one is truly familiar with, even if you do try to revi…

Yeah, it's the "eager apprentice" problem, common almost everywhere. Solution is to make them stop and double-check before running ahead, in software development, concise design documents outlining what the problem is, what possible solutions are and what the chosen solution is, and why, then review this together with the person, before they can move on to implement it.

This particular apprentice is also my boss, an overall reasonable guy and has more experience in the software industry than myself, so there's that. He's just not a developer.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#69

Post by amluto »

stingraycharles wrote: post #63
ZephyrBlu wrote: post #43

Earlier quoted context omitted.

Thinking of AI-generated code and AI code reviews as the same "AI" is not correct. The reviewer is using a fresh context window with no previous knowledge of the changes. That is why it's powerful, because you get the agent to interrogate the code without any preconception about the changes. I found the Devin reviewer to be very good, and have heard good things about Cursor's Bugbot. I've also found asking an agent w…

I’m not saying that it’s not useful, I’m saying that it’s not useful in a “human in the loop” situation. This type of AI-to-AI review should be done agent-to-agent, not through Github PRs with tools like Devin. In a manual review, I then expect all “machinery” to already be properly reviewed, and can focus on design / architecture. I would like an AI assisted review tool to make that part easier, not do the actual re…

I’m not convinced by the automatic agent-to-agent thing. I find that, if I manually ask a standard harness “Review a..b, individually and for combined effect”, I get some mix of catching genuine errors (some quite deep), incorrect flags where the correct course of action is to ignore them or modify the commit messages, and comments where the correct course of action is to think deeply.

If I were to automate the back and forth, I would get spurious changes that “fix” what wasn’t broken and a removal of the actual interesting bits.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#70

Post by fabianlindfors »

sdeframond wrote: post #67
fabianlindfors wrote: post #49

Earlier quoted context omitted.

Firecracker, with some tiny modifications to better manage memory for the deep nesting of forks

Would you mind sharing your infra budget needed to spin these VMs ? Surely it is reasonable, but also way more than our budget. Id like to compare.

Sure. It's a bit hard to quantify because we need to run these on bare-metal machines and the unit cost is pretty high.

We run our test workload as well as a few other startups' that we have onboarded on one AWS ARM bare-metal machine at $1.7k a month. We don't saturate that machine fully either so I'm not really sure what the amortized cost would be. Certainly more expensive than Github Actions but not by a crazy amount, and the value we get out of it is way higher than GA.

Post reply on HN