Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.6 Luna vs. GPT-6 Astra: Is a $1.20 Model Good Enough for Code Review?

entelligence.ai

Post reply on HN
Print view View original

101–110 of 120 posts

Re: GPT-5.6 Luna vs. GPT-6 Astra: Is a $1.20 Model Good Enough for Code Review?

#101

Post by jamienicol »

epolanski wrote: post #83
nonethewiser wrote: post #18

AI should be used for code review but not in CI. You should already have 2+ developers looking at most PRs. And these developers should absolutely use AI. The PR author should use AI. But what you should not do is pipe the AI output directly into the PR and tell the PR author to deal with it. That's adding noise to the PR review process. Everything it says is something the PR author needs to validate as relevant, hel…

I'm glad I work in places where there's no such silly pointless rules like how many people need to review a PR. The PR author asks for feedback if it needs feedback, otherwise it merges it, period. I don't know why and when the world got convinced that all this bureaucracy is a "best practice", when it's just a practice, that can be good, or a waste of time depending on many factors. Like do doctors when analyzing yo…

My partner gave birth last month and the hospital had a system where every hour a second midwife would come into the room and look at all the charts and review their colleague’s work with a second, fresh pair of eyes.

Re: GPT-5.6 Luna vs. GPT-6 Astra: Is a $1.20 Model Good Enough for Code Review?

#102

Post by theplumber »

It would be great if all these benchmarks would also provide a slop metric. The main issue I see is not finding bugs but reporting “stupid” bugs and and if the recommendations are followed you end up overengineering the wrong things basically producing AI slop. GPT is prime candidate for this. Fable as well but less than GPT

Re: GPT-5.6 Luna vs. GPT-6 Astra: Is a $1.20 Model Good Enough for Code Review?

#103

Post by theplumber »

holoduke wrote: post #92
stefangordon wrote: post #86

Earlier quoted context omitted.

Our agents automatically review our PR's - the authors agents automatically see the feedback and make fixes, and automatically merge when everything is green. A well authored CI review process is significantly better than any human could do. We have the AI review not only the changes but clone and investigate all related repositories that integrate with the code to evaluate interactions, check all open and historical…

Must be a token burner setup. Nice if you really got it to work. Now also add business agents that come up with new features based on actual research. Your automated humanless company is almost there.

Sounds like an AI-OS slop machine…been there, done that!

Re: GPT-5.6 Luna vs. GPT-6 Astra: Is a $1.20 Model Good Enough for Code Review?

#104

Post by sick_of_slop »

holoduke wrote: post #92
stefangordon wrote: post #86

Earlier quoted context omitted.

Our agents automatically review our PR's - the authors agents automatically see the feedback and make fixes, and automatically merge when everything is green. A well authored CI review process is significantly better than any human could do. We have the AI review not only the changes but clone and investigate all related repositories that integrate with the code to evaluate interactions, check all open and historical…

Must be a token burner setup. Nice if you really got it to work. Now also add business agents that come up with new features based on actual research. Your automated humanless company is almost there.

[dead]

Re: GPT-5.6 Luna vs. GPT-6 Astra: Is a $1.20 Model Good Enough for Code Review?

#105

Post by sick_of_slop »

epolanski wrote: post #85
MattDamonSpace wrote: post #29

Earlier quoted context omitted.

Yes. I sympathize with the “every PR needs human eyes on it” ethos but fundamentally the only thing that scales with AI PR generation is AI PR review.

I don't get the point of AI pr review. If the AI can review the pr, it can do so before the PR exists in the first place. It's a tool/skill/prompt. Just share it with the end user or encode it in the codebase in some files. It may actually be even more useful as that person will use it often and improve it or can it.

[dead]

Re: GPT-5.6 Luna vs. GPT-6 Astra: Is a $1.20 Model Good Enough for Code Review?

#106

Post by jbellis »

For the mj adversarial review we use a configurable coordinator (I recommend Fable) with six subagents (I recommend Luna) for specific specialties like code complexity. Narrowing down what a smaller model like Luna needs to look for / care about helps them do better work; then the larger model synthesizes and fills in any gaps.

https://blog.brokk.ai/mjolnir-automated-cross-vendor-adversa...

Re: GPT-5.6 Luna vs. GPT-6 Astra: Is a $1.20 Model Good Enough for Code Review?

#107

Post by avadodin »

nvch wrote: post #99
avadodin wrote: post #98

Any model is good enough. Even tiny local models can provide some value and their false positive rate is still relatively low and warrants a proper reply.

With tiny models, we're getting into the territory of horoscopes and divination. While it is possible for a sentient being to derive value by using them as a random seed for thinking, the value is produced by something different from the seed.

Have you tried and how do they compare to human models as in the average human in a review?

I wouldn't put a lot of weight into an all-clear result but if Ling tiny tells me my function lacks an essential check it will force me to consider it seriously or to explicitly clarify that such checks are not relevant.

Re: GPT-5.6 Luna vs. GPT-6 Astra: Is a $1.20 Model Good Enough for Code Review?

#108

Post by samusiam »

gregwebs wrote: post #57

They state Luna is good enough, but its accuracy of findings is 74% whereas Astra is 96%. Dealing with false positives is expensive. I am finding AI doing its own reviews as part of the process to be the key to productivity. I do subagent (fresh context reviews) at multiple stages with well-specified review criteria. It is really expensive to do with OpenAI or Claude API billing. Deepseek or the discounted monthly pl…

My thoughts exactly. That 22pp gap is massive, and it seems to me Astra is worth every extra penny.

Re: GPT-5.6 Luna vs. GPT-6 Astra: Is a $1.20 Model Good Enough for Code Review?

#109

Post by samusiam »

ltbarcly3 wrote: post #46

They seem to conclude Luna is a better value, but their analysis is dumb. They just break it down to $/bug found. However, Luna missed 23 bugs that Astra found, and identified 24 bugs that weren't really bugs. That's horrible. Astra had 96% precision. The cost to care about here isn't just how much it costs to run the code review, or the cost per true-positive. It's the cost of dealing with this system. A code review…

To me, one interesting piece of analysis was whether Luna had benefit on top of Astra -- i.e., running both and synthesizing their findings. But even with that, it raises the question whether running a second Astra pass, or Sol, or even a model from another family (GLM? Fable?) would deliver even more additive benefit.

Re: GPT-5.6 Luna vs. GPT-6 Astra: Is a $1.20 Model Good Enough for Code Review?

#110

Post by mooman219 »

StevenWaterman wrote: post #4

$0.10 extra per pr review is nothing. What software company is willing to accept worse reviews and less bugs found to save 10 cents?

Worse seems subjective here. It seems Luna found bugs Astra did not, and vice versa. Astra had lower noise overall. I think my take away here is to use a blend of models given their different abilities to find different domains of bugs.
Post reply on HN