Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-6 Astra in code review: Gains, privacy, and cost

coderabbit.ai

Post reply on HN
Print view View original

31–40 of 77 posts

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#31

Post by bushbaba »

sdeframond wrote: post #28

How do you guys review AI-generated code ? In our team, frontend work is vibe-coded by the PO and merged as-is without review. Backend is coded by developers, using AI but in a slower, more controlled way. Recently, our PO has been trying his hand at vibe-coding the backend. I must say he is a smart guy, almost technical but not quite a developer. We've just been handed a burst of stacked PRs amounting for ~15k LOC b…

I have AI confirm the logic works as expected, but review for system design.

Often in both web/backend I’ve found AI to produce overly duplicative code, or have aspects that could be hard to maintain. Generally less due to the AI, and more because of the prompt itself.

That and even if you’re going to AI slop it up, I’d still demand it be broken up into 1-2k LOC chunks or per meaningful “thing”. This also lets us gradually ramp the change to confirm it actually works earlier on

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#32

Post by yomismoaqui »

sdeframond wrote: post #28

How do you guys review AI-generated code ? In our team, frontend work is vibe-coded by the PO and merged as-is without review. Backend is coded by developers, using AI but in a slower, more controlled way. Recently, our PO has been trying his hand at vibe-coding the backend. I must say he is a smart guy, almost technical but not quite a developer. We've just been handed a burst of stacked PRs amounting for ~15k LOC b…

Invest in having a good test suite that validates the functionality introduced by that code. Also AI can review code in an adversarial way and apply those fixes (that ideally will keep the previous tests you did on green)

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#33

Post by fabianlindfors »

sdeframond wrote: post #28

How do you guys review AI-generated code ? In our team, frontend work is vibe-coded by the PO and merged as-is without review. Backend is coded by developers, using AI but in a slower, more controlled way. Recently, our PO has been trying his hand at vibe-coding the backend. I must say he is a smart guy, almost technical but not quite a developer. We've just been handed a burst of stacked PRs amounting for ~15k LOC b…

We try to avoid reviewing AI-generated code and built our own testing framework and platform to make that possible.

Our principle is that our tests should give us enough confidence to not have to look at the code (which ends up being true for most changes we make). The core thing that makes this possible is that we run our entire code and infra (including fakes for external dependencies) in isolated, forkable environments and write tests against that, so they are as E2E as can possibly be.

The problem then shifts from reviewing code to reviewing tests and that's why we built our own platform. We have a UI that can diff tests, so we know what changed, and a visual way to inspect what the tests actually did. A test could drive a browser like a user would, and in our UI we get a replay of that browser interaction to look at. The browser is talking to a real version of our backend, and the tests can perform assertions against the database and fakes and really anything in our system.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#34

Post by Sharlin »

bitlad wrote: post #24
eyalitki wrote: post #7

Comparison was done in the scope of coderabbit AI code review tool, which sadly makes it practically irrelevant. My personal experience as a software engineer, and a former security researcher who did manual code audit, is that this code review tool has such poor results that it isn't worth the "noise" and friction it causes developers during C/I code review

It does add lot of noise after a point you start ignoring the suggestions and findings. Code generated these days with fable and sol are near perfect. What issues they might have is logical errors.

You have a very interesting definition of "perfect", then.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#35

Post by rhdunn »

pinkgolem wrote: post #14
stingraycharles wrote: post #9

Earlier quoted context omitted.

Yeah I personally don’t understand the point of AI code review tools all that much, as AI is already generating the code as well. All of these AI code review tools create so much noise, yet don’t catch the really important things.

What really important things are human reviews catching in your org? I just feel more and more like the effort invested in manual reviews is not worth it

1. Does the implementation fit in the architecture/style of the project?

2. Are there potential security, accessibility, performance, etc. issues?

3. Domain specific knowledge (SQL, ASP.NET, XQuery, etc.) where there are better ways of solving a problem, or possible issues not handled.

4. Sense checking ... is the code easy to read? does it need an explanatory comment? does it need named parameters? etc.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#36

Post by nkmnz »

stingraycharles wrote: post #9
eyalitki wrote: post #7

Comparison was done in the scope of coderabbit AI code review tool, which sadly makes it practically irrelevant. My personal experience as a software engineer, and a former security researcher who did manual code audit, is that this code review tool has such poor results that it isn't worth the "noise" and friction it causes developers during C/I code review

Yeah I personally don’t understand the point of AI code review tools all that much, as AI is already generating the code as well. All of these AI code review tools create so much noise, yet don’t catch the really important things.

The noise is a huge problem, indeed. Still, a panel of review agents using models and harnesses different from the one implementing a set of changes has proven immensely useful for myself. The panel is basically an n×m matrix of agents and highly specific review prompts, i.e.:

- review for intent fulfillment: is the ticket done?

- review for correctness: race condition bugs, ...

- review for security: check against this list of sources and best practices

- review for api conformity: identify all surfaces of systems outside this codebase touched by the code changes and check against their docs

- etc. pp., same for maintainability, observability & analytics, test coverage, usage of feature flags

The matrix is sparse, so not every model is used with each of the review categories. Effort levels vary, too. The next stage does a consolidation across all findings, then another stage spins up one agent per finding and investigates the whole codebases for identical / similar instances of the finding; finally, it suggests a fix.

This works extremely well for finding deficits, but the amount of noise drives me insane, too. Lots of feedback is technically correct and "by the book", but pretty useless in practical terms – or even detrimental because the amount of code written and thus the size of the change set explodes. I'm not yet sure how to tackle this problem, any suggestions are welcome!

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#37

Post by eterps »

fabianlindfors wrote: post #33
sdeframond wrote: post #28

How do you guys review AI-generated code ? In our team, frontend work is vibe-coded by the PO and merged as-is without review. Backend is coded by developers, using AI but in a slower, more controlled way. Recently, our PO has been trying his hand at vibe-coding the backend. I must say he is a smart guy, almost technical but not quite a developer. We've just been handed a burst of stacked PRs amounting for ~15k LOC b…

We try to avoid reviewing AI-generated code and built our own testing framework and platform to make that possible. Our principle is that our tests should give us enough confidence to not have to look at the code (which ends up being true for most changes we make). The core thing that makes this possible is that we run our entire code and infra (including fakes for external dependencies) in isolated, forkable environ…

But with full blown e2e browser tests the test suite duration can go through the roof. How do you deal with that?

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#38

Post by sdeframond »

fabianlindfors wrote: post #33
sdeframond wrote: post #28

How do you guys review AI-generated code ? In our team, frontend work is vibe-coded by the PO and merged as-is without review. Backend is coded by developers, using AI but in a slower, more controlled way. Recently, our PO has been trying his hand at vibe-coding the backend. I must say he is a smart guy, almost technical but not quite a developer. We've just been handed a burst of stacked PRs amounting for ~15k LOC b…

We try to avoid reviewing AI-generated code and built our own testing framework and platform to make that possible. Our principle is that our tests should give us enough confidence to not have to look at the code (which ends up being true for most changes we make). The core thing that makes this possible is that we run our entire code and infra (including fakes for external dependencies) in isolated, forkable environ…

Well, (AI-generated) test are about half of these PRs' code. So that's still ~8k lines to review...

What techno/service did you base your framework on? How long did it take to set it up? How many are you?

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#39

Post by embedding-shape »

mobelkh wrote: post #30
sdeframond wrote: post #28

How do you guys review AI-generated code ? In our team, frontend work is vibe-coded by the PO and merged as-is without review. Backend is coded by developers, using AI but in a slower, more controlled way. Recently, our PO has been trying his hand at vibe-coding the backend. I must say he is a smart guy, almost technical but not quite a developer. We've just been handed a burst of stacked PRs amounting for ~15k LOC b…

throw his garbage out, the time and effort taken to review that is magnitudes more than what it took to prompt it. have him start with an overall design doc if his change is 15k, it's definitely worth a design doc. and then have his contributions reviewed in pieces of 200-300 LoC PRs. any other solution is trading stability and system knowledge, that's 15k LoC no one is truly familiar with, even if you do try to revi…

Yeah, it's the "eager apprentice" problem, common almost everywhere. Solution is to make them stop and double-check before running ahead, in software development, concise design documents outlining what the problem is, what possible solutions are and what the chosen solution is, and why, then review this together with the person, before they can move on to implement it.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#40

Post by spockz »

stingraycharles wrote: post #9
eyalitki wrote: post #7

Comparison was done in the scope of coderabbit AI code review tool, which sadly makes it practically irrelevant. My personal experience as a software engineer, and a former security researcher who did manual code audit, is that this code review tool has such poor results that it isn't worth the "noise" and friction it causes developers during C/I code review

Yeah I personally don’t understand the point of AI code review tools all that much, as AI is already generating the code as well. All of these AI code review tools create so much noise, yet don’t catch the really important things.

Where I find it shines it to find inconsistencies. My readme or docs or ADR something should work like X but it finds a test where it tests something different and the test is green. Or other similar drift.

Yes, your prompt need to include to look for certain “quality” aspects you care about. But once that is there it can help find a lot of things.

It can also help in finding edge cases. It is really about the prompt.

Post reply on HN