Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-6 Astra in code review: Gains, privacy, and cost

coderabbit.ai

Post reply on HN
Print view View original

41–50 of 77 posts

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#41

Post by fabianlindfors »

eterps wrote: post #37
fabianlindfors wrote: post #33

Earlier quoted context omitted.

We try to avoid reviewing AI-generated code and built our own testing framework and platform to make that possible. Our principle is that our tests should give us enough confidence to not have to look at the code (which ends up being true for most changes we make). The core thing that makes this possible is that we run our entire code and infra (including fakes for external dependencies) in isolated, forkable environ…

But with full blown e2e browser tests the test suite duration can go through the roof. How do you deal with that?

Forking!

We run the entire stack (browser, frontend, backend, database, etc) in a Linux VM, so latency between each of the pieces is as tiny as can be. This is quite different from "standard" E2E tests I've seen where the test browsers uses something like a persistent staging environment.

The real key is that we can fork that entire Linux VM to take different paths down our testing scenarios, and can run multiple of them in parallel. Tests may look something like:

  new user signs up:
  |- creates a todo
     |- ...
     |- ...
  |- creates a list
The two nested tests then start from the exact same point, where the previous test left off, but can run in parallel. With enough hardware, the full suite will run as fast as the slowest branch of the test tree. When we switched away from our previous integration test suite to this (not E2E), our tests actually became faster because they share setup through the forking.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#42

Post by fabianlindfors »

sdeframond wrote: post #38
fabianlindfors wrote: post #33

Earlier quoted context omitted.

We try to avoid reviewing AI-generated code and built our own testing framework and platform to make that possible. Our principle is that our tests should give us enough confidence to not have to look at the code (which ends up being true for most changes we make). The core thing that makes this possible is that we run our entire code and infra (including fakes for external dependencies) in isolated, forkable environ…

Well, (AI-generated) test are about half of these PRs' code. So that's still ~8k lines to review... What techno/service did you base your framework on? How long did it take to set it up? How many are you?

That's the point, we don't review the test code either. Our platform gives us a UI for inspecting not the test code but what actually happened during the test. Like a browser replay, the results of a database query, assertions against those, etc.

This is much more information dense than something like the tests and is a representation of what actually happened during the tests, rather than what the test itself did (which I agree sucks to review, especially AI-generated).

The framework is our own that bundles/adapts some familiar components: Jest-like asssertions, Playwright browser API, typed database client from Bun, Kubernetes client, etc. The tests are written in Typescript but the main code doesn't have to be (just runs containerized in the environment).

We've been building the platform and using it continuously since May but setting it up on a new project takes like 1-2 days of largely autonomous coding agent work. We are just two engineers on our team but have been onboarding other startups to the platform recently so there are a few different teams using it now for their own codebases. It's fully generic so works for any infra or stack.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#43

Post by ZephyrBlu »

stingraycharles wrote: post #9
eyalitki wrote: post #7

Comparison was done in the scope of coderabbit AI code review tool, which sadly makes it practically irrelevant. My personal experience as a software engineer, and a former security researcher who did manual code audit, is that this code review tool has such poor results that it isn't worth the "noise" and friction it causes developers during C/I code review

Yeah I personally don’t understand the point of AI code review tools all that much, as AI is already generating the code as well. All of these AI code review tools create so much noise, yet don’t catch the really important things.

Thinking of AI-generated code and AI code reviews as the same "AI" is not correct. The reviewer is using a fresh context window with no previous knowledge of the changes. That is why it's powerful, because you get the agent to interrogate the code without any preconception about the changes.

I found the Devin reviewer to be very good, and have heard good things about Cursor's Bugbot. I've also found asking an agent with fresh context or subagent to adversarially review locally is good.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#44

Post by ninkendo »

sdeframond wrote: post #28

How do you guys review AI-generated code ? In our team, frontend work is vibe-coded by the PO and merged as-is without review. Backend is coded by developers, using AI but in a slower, more controlled way. Recently, our PO has been trying his hand at vibe-coding the backend. I must say he is a smart guy, almost technical but not quite a developer. We've just been handed a burst of stacked PRs amounting for ~15k LOC b…

The answer to this is gonna vary wildly depending on what kind of codebase it is.

A large, mature codebase that predates LLM’s and for which changes need a high level of scrutiny regardless of who made them (think llvm, WebKit, important foundational software), you’re going to want humans in the loop as much as ever… I think reviewing LLM output is the most important thing a human can provide.

But for vibe coded apps where you can just one-shot another one if anything goes wrong? Just vibe the reviews too, who cares. Let the robots review the robots.

Be careful with doing AI review if your codebase is in the former category. Or your codebase will quickly turn into the latter. Complete with “you can just one-shot another one”, because if nobody understands the code any more, there’s not much lost by just you (or your competitors, etc) replacing it wholesale with an AI-written alternative.

I struggle with this a lot. 2 years ago we had a half dozen PR’s a day with a lot of careful review, and now there’s more like 30 of them per day and most people are just rubber stamping them after the AI reviews it. I’m still fighting the good fight trying to review every line of the PR’s I have time to look at, but that constitutes maybe 10% of them. Not only am I barely making a dent, but it’s awkward when I post nitpicks like “this function should go in this module”, etc, the author usually looks at me funny like “why are you even reading this”. Our codebase is gradually becoming more and more vibe coded, and it’s depressing me.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#46

Post by tosh »

SneakyZero wrote: post #11

Astra seems to be really slow. Maybe it intends to read more context. But from my experience it is definitely slower than 5.6 sol when handling same tasks.

fwiw I found Astra to be faster than Sol w/ both on medium reasoning for simple agentic coding

difficult to compare though because for more open ended, complex tasks Sol might miss something that Astra notices and then Sol might yield a cheaper but worse outcome

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#47

Post by skrellm »

villish wrote: post #18
ramon156 wrote: post #4

Both OAI and Anthropic seem to have released a model that is slightly better but cost ~2x the previous iteration. Interesting play

That likely won’t change if other competitors don’t take the lead at some point. If companies are willing to pay top dollar for the best models AND they get to extract as much money from Chinese labs distilling Astra/Fable it makes no sense to lower prices. Obviously not great for everyday users who don’t have unlimited money.

> If companies are willing to pay top dollar for the best models

But they are not willing to pay. https://news.ycombinator.com/item?id=49566137

The referenced NYT article says Open Source LLM's market share increased to 58% from yesteryear's 10%.

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#48

Post by eterps »

fabianlindfors wrote: post #41
eterps wrote: post #37

Earlier quoted context omitted.

But with full blown e2e browser tests the test suite duration can go through the roof. How do you deal with that?

Forking! We run the entire stack (browser, frontend, backend, database, etc) in a Linux VM, so latency between each of the pieces is as tiny as can be. This is quite different from "standard" E2E tests I've seen where the test browsers uses something like a persistent staging environment. The real key is that we can fork that entire Linux VM to take different paths down our testing scenarios, and can run multiple of…

Which VM technology do you use?

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#49

Post by fabianlindfors »

eterps wrote: post #48
fabianlindfors wrote: post #41

Earlier quoted context omitted.

Forking! We run the entire stack (browser, frontend, backend, database, etc) in a Linux VM, so latency between each of the pieces is as tiny as can be. This is quite different from "standard" E2E tests I've seen where the test browsers uses something like a persistent staging environment. The real key is that we can fork that entire Linux VM to take different paths down our testing scenarios, and can run multiple of…

Which VM technology do you use?

Firecracker, with some tiny modifications to better manage memory for the deep nesting of forks

Re: GPT-6 Astra in code review: Gains, privacy, and cost

#50

Post by 9dev »

embedding-shape wrote: post #27
9dev wrote: post #19

Earlier quoted context omitted.

All of these are angles an AI reviewer can test for as well, and will (IME) mostly catch mistakes correctly. I also still manually review code, and usually also catch issues, but the severity of what I find shrinks ever further as agents get better. The sprawling code comments are becoming the most draining part of code review though, that's really killing me from the inside.

> All of these are angles an AI reviewer can test for as well, and will (IME) mostly catch mistakes correctly. No, none of today's AI would give you enough signal around "should this thing be built in the first place" nor if it's the correct solution on a high level. They don't understand why you are doing what you are doing, and even if you explain it, they still don't actually understand the motivation and lots of…

I don’t think I claimed agent reviews to be a panacea. It’s a tool that can help you lower the review pressure in companies working with agentic coding tools.
Post reply on HN