Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If you are good at code review, you will be good at using AI agents

seangoedecke.com

Post reply on HN
Print view View original

171–180 of 203 posts

Re: If you are good at code review, you will be good at using AI agents

#171

Post by rsynnott »

nkmnz wrote: post #37
rsynnott wrote: post #23

This idea that you can get good results from a bad process as long as you have good quality control seems… dubious, to say the least. “Sure, it’ll produce endless broken nonsense, but as long as someone is checking, it’s fine.” This, generally, doesn’t really work. You see people _try_ it in industry a bit; have a process which produces a high rate of failures, catch them in QA, rework (the US car industry used to be…

> This idea that you can get good results from a bad process This idea is called "evolution"... > as long as you have good quality control ...and it's QA is death on every single level of the systems: cell, organism, species, and ecosystem. You must consider that those devs or companies with not-good-enough QA will end up dead (from a business perspective).

I look forward to software which takes several million years to produce and tends to die of Software Cancer.

Like, evolution is not _good_ at ‘designing’ things.

Re: If you are good at code review, you will be good at using AI agents

#172

Post by btown »

Manfred wrote: post #48
rsynnott wrote: post #23

This idea that you can get good results from a bad process as long as you have good quality control seems… dubious, to say the least. “Sure, it’ll produce endless broken nonsense, but as long as someone is checking, it’s fine.” This, generally, doesn’t really work. You see people _try_ it in industry a bit; have a process which produces a high rate of failures, catch them in QA, rework (the US car industry used to be…

Reviewing code from less experienced or unmotivated people is also very taxing, both in a cognitive and emotional sense. It will never approach a really good level of quality because you just give up after 4 rounds of reviews on the same feature.

Here’s the thing about AI though - you don’t need to worry about its confidence or impact on professional development if you’re overly critical, and it will do a turn within seconds. That gives a tremendous amount of flexibility and leverage to the code reviewer. Works better on some types of problems than others, but it’s worth exploring!

Re: If you are good at code review, you will be good at using AI agents

#173

Post by shepherdjerred »

EdwardDiego wrote: post #145
Manfred wrote: post #48

Earlier quoted context omitted.

Reviewing code from less experienced or unmotivated people is also very taxing, both in a cognitive and emotional sense. It will never approach a really good level of quality because you just give up after 4 rounds of reviews on the same feature.

Except humans learn from your PR comments and in other interactions with more experienced people, and so inexperienced devs become experienced devs eventually. LLMs are not so trainable.

LLMs can learn if you provide it rules in your repo, and update those rules as you identify the common mistakes the LLM makes

Re: If you are good at code review, you will be good at using AI agents

#174

Post by adastra22 »

jmull wrote: post #49
bcrosby95 wrote: post #28

Earlier quoted context omitted.

I don't think it's that simple. Fundamentally, unit tests are using the same system to write your invariants twice, it just so happens that they're different enough that failure in one tends to reveal a bug in another. You can't reasonably state this won't be the case with tools built for code review until the failure cases are examined. Furthermore a simple way to help get around this is by writing code with one pro…

> unit tests are using the same system to write your invariants twice For unit tests, the parts of the system that are the same are not under test, while the parts that are different are under test. The problem with using AI to review AI is that what you're checking is the same as what you're checking it with. Checking the output of one LLM with another brand probably helps, but they may also have a lot of similariti…

> The problem with using AI to review AI is that what you're checking is the same as what you're checking it with.

This isn't true. Every instantiation of the LLM is different. Oversimplifying a little, but hallucination emerges when low-probability next words are selected. True explanations, on the other hand, act as attractors in state-space. Once stumbled upon, they are consistently preserved.

So run a bunch of LLM instances in parallel with the same prompt. The built-in randomness & temperature settings will ensure you get many different answers, some quite crazy. Evaluate them in new LLM instances with fresh context. In just 1-2 iterations you will hone in on state-space attractors, which are chains of reasoning well supported by the training set.

Re: If you are good at code review, you will be good at using AI agents

#175

Post by org3 »

EdwardDiego wrote: post #145
Manfred wrote: post #48

Earlier quoted context omitted.

Reviewing code from less experienced or unmotivated people is also very taxing, both in a cognitive and emotional sense. It will never approach a really good level of quality because you just give up after 4 rounds of reviews on the same feature.

Except humans learn from your PR comments and in other interactions with more experienced people, and so inexperienced devs become experienced devs eventually. LLMs are not so trainable.

Some people say we're near the end of pre-training scaling, and RLHF etc is going to be more important in the future. I'm interested in trying out systems like https://github.com/OpenPipe/ART to be able to train agents to work on a particular codebase and learn from my development logs and previous interactions with agents.

Re: If you are good at code review, you will be good at using AI agents

#176

Post by coffeefirst »

scuff3d wrote: post #149
coffeefirst wrote: post #98

Earlier quoted context omitted.

Yeah. I'm not even anti-LM. Little things—research, "write TS types for this object", search my codebase, go figure out exactly what line in the Django rest framework is causing this weird behavior, —are working great and saving me an hour here and 15m there. It's really obvious when people lean on it, because they don't act like a beginner (trying things that might not work) or just being sloppy (where there's a log…

It's also nice for handling some little tasks that otherwise would have been just annoying enough for me to not do it. Small 5 or 6 line functions that I would have had to fiddle with for far longer to get right.

Totally. I have to watch it like a hawk though. I've had it get the logic backwards on trivial "copy paste this from from the docs" functions.

Re: If you are good at code review, you will be good at using AI agents

#177

Post by theshrike79 »

kiitos wrote: post #127
jmcodes wrote: post #55

Earlier quoted context omitted.

Maybe I'm weird but I don't actually enjoy the act of _writing_ code. I enjoy problem solving and creating something. I enjoy decomposing systems and putting them back together in a better state, but actually manually typing out code isn't something I enjoy. When I use an LLM to code I feel like I can go from idea to something I can work with in much less time than I would have normally. Our codebase is more type-saf…

the time spent literally typing code into an editor is never the bottleneck in any competently-run project if the act of writing code is something you consider a burden rather than a joy then my friend you are in the wrong profession

But it's not exactly rewarding to add one more CRUD endpoint. It's a shit-ton of typing in multiple layers.

An LLM can do it in two minutes while I fetch coffee, then I can proceed to add the complex bits (if there are any)

Re: If you are good at code review, you will be good at using AI agents

#178

Post by estimator7292 »

rsynnott wrote: post #23

This idea that you can get good results from a bad process as long as you have good quality control seems… dubious, to say the least. “Sure, it’ll produce endless broken nonsense, but as long as someone is checking, it’s fine.” This, generally, doesn’t really work. You see people _try_ it in industry a bit; have a process which produces a high rate of failures, catch them in QA, rework (the US car industry used to be…

Imagine a factory making injection molded plastic toys but instead of pumping out perfect parts 99.999% of the time, the machine gives you 50% and you have to pay people to pull out the bad ones from a full speed assembly line and hope no bad ones get through.

Re: If you are good at code review, you will be good at using AI agents

#179

Post by hakfoo »

Manfred wrote: post #48
rsynnott wrote: post #23

This idea that you can get good results from a bad process as long as you have good quality control seems… dubious, to say the least. “Sure, it’ll produce endless broken nonsense, but as long as someone is checking, it’s fine.” This, generally, doesn’t really work. You see people _try_ it in industry a bit; have a process which produces a high rate of failures, catch them in QA, rework (the US car industry used to be…

Reviewing code from less experienced or unmotivated people is also very taxing, both in a cognitive and emotional sense. It will never approach a really good level of quality because you just give up after 4 rounds of reviews on the same feature.

With human co-workers, you can generally assume things you can't with AI.

My human co-workers generally have good faith. Even the developer who was clearly on the verge of getting a role elsewhere without his heart in it-- he tried to solve the problems assigned to him, not some random delusion that the words happened to echo. I don't have that level of trust with AI.

If there's a misunderstanding the problem or the context, it's probably still the product of a recognizable logic flow that you can use to discuss what went wrong. I can ask Claude "Why are you converting this amount from Serbian Dinars to Poppyseed Bagels in line 476?" but will its answer be meaningful?

Human code review often involves a bit of a shared background. We've been working with the same codebases for several years, so we're going to use existing conventions. In this situation, the "AI knows all and sees all" becomes an anti-feature-- it may optimize for "this is how most people solve this task from a blank slate" rather than "it's less of a cognitive burden for the overall process if your single change is consistent with 500 other similar structures which have been in place since the Clinton administration."

There may be ways to try to force-feed AI this behaviour, but the more effort you devote to priming and pre-configuring the machine, the less you're actually saving over doing the actual work in the first place.

Re: If you are good at code review, you will be good at using AI agents

#180

Post by yen223 »

estimator7292 wrote: post #178
rsynnott wrote: post #23

This idea that you can get good results from a bad process as long as you have good quality control seems… dubious, to say the least. “Sure, it’ll produce endless broken nonsense, but as long as someone is checking, it’s fine.” This, generally, doesn’t really work. You see people _try_ it in industry a bit; have a process which produces a high rate of failures, catch them in QA, rework (the US car industry used to be…

Imagine a factory making injection molded plastic toys but instead of pumping out perfect parts 99.999% of the time, the machine gives you 50% and you have to pay people to pull out the bad ones from a full speed assembly line and hope no bad ones get through.

Is this not how microchips are made?
Post reply on HN