Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If you are good at code review, you will be good at using AI agents

seangoedecke.com

Post reply on HN
Print view View original

111–120 of 203 posts

Re: If you are good at code review, you will be good at using AI agents

#111

Post by HarHarVeryFunny »

ben_w wrote: post #86
rsynnott wrote: post #23

This idea that you can get good results from a bad process as long as you have good quality control seems… dubious, to say the least. “Sure, it’ll produce endless broken nonsense, but as long as someone is checking, it’s fine.” This, generally, doesn’t really work. You see people _try_ it in industry a bit; have a process which produces a high rate of failures, catch them in QA, rework (the US car industry used to be…

> I don’t know of any case where it has really worked out. Supermarket vegetables.

Are you saying that supermarket vegetables/produce are good?

Quite a bit of it, like Tomatoes and Strawberries, is just crap. Form over substance. Nice color and zero flavor. Selected for delivery/shelf-life/appearance rather actually being any good.

Re: If you are good at code review, you will be good at using AI agents

#112

Post by dwaltrip »

bluefirebrand wrote: post #85
nkmnz wrote: post #37

Earlier quoted context omitted.

> This idea that you can get good results from a bad process This idea is called "evolution"... > as long as you have good quality control ...and it's QA is death on every single level of the systems: cell, organism, species, and ecosystem. You must consider that those devs or companies with not-good-enough QA will end up dead (from a business perspective).

So we're software evolvers now, not engineers? Sounds like a stupid path forward to me

We’ve always been software evolvers. Ideas that have been around for decades, such as “a codebase is a garden”, are even more relevant now.

Re: If you are good at code review, you will be good at using AI agents

#113

Post by ben_w »

vrighter wrote: post #109
ben_w wrote: post #82

Earlier quoted context omitted.

Weirdly, you can not only do this, it somehow does actually catch some of its own mistakes. Not all of the mistakes, they generally still have a performance ceiling less than human experts (though even this disclaimer is still simplifying), but this kind of self-critique is basically what makes the early "reasoning" models one up over simple chat models: for the first-n :END: tokens, replace with "wait" and see it at…

the "pick something usually better" sounds a lot like "and then draw the rest of the f*** owl"

Turned out that for a lot of things (not all things, Transformers have a lot of weaknesses), using a neural network to score an output is, if not "fine", then at least "ok".

Generating 10 options with mediocre mean and some standard deviation, and then evaluating which is best, is much easier than deliberative reasoning to just get one thing right in the first place more often.

Re: If you are good at code review, you will be good at using AI agents

#114

Post by ben_w »

HarHarVeryFunny wrote: post #111
ben_w wrote: post #86

Earlier quoted context omitted.

> I don’t know of any case where it has really worked out. Supermarket vegetables.

Are you saying that supermarket vegetables/produce are good? Quite a bit of it, like Tomatoes and Strawberries, is just crap. Form over substance. Nice color and zero flavor. Selected for delivery/shelf-life/appearance rather actually being any good.

> Form over substance. Nice color and zero flavor. Selected for delivery/shelf-life/appearance rather actually being any good.

From an economics POV, that's the correct test.

I was also considering the way the US food standards allows a lot of insect parts in the products, but wasn't sure how to phrase it.

Re: If you are good at code review, you will be good at using AI agents

#115

Post by keeda »

rsynnott wrote: post #23

This idea that you can get good results from a bad process as long as you have good quality control seems… dubious, to say the least. “Sure, it’ll produce endless broken nonsense, but as long as someone is checking, it’s fine.” This, generally, doesn’t really work. You see people _try_ it in industry a bit; have a process which produces a high rate of failures, catch them in QA, rework (the US car industry used to be…

1. The flaw in this premise is that the process is bad. Aside from the countless anecdotal reports about how AI and agents are improving productivity, there are actual studies showing 25 - 55% boosts. Yes, RCTs at larger size than the METR one that keeps getting bandied about: https://news.ycombinator.com/item?id=44860577 and many more on Google Scholar: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&as_ylo...

2. Quality control is key to good processes as well. Code review is literally a best practice in the software industry. Especially in BigTech and high-performing organizations. That is, even for humans, including those that could be considered the cream of the industry, code review is a standard step of the delivery process.

3. People have posted their GitHub profiles and projects (including on this very forum) to show how AI is working out for them. Browse through some of them and see how much "endless broken nonsense" you find. And if that seems unscientific, well go back to point 1.

Re: If you are good at code review, you will be good at using AI agents

#116

Post by [deleted] »

gus_massa wrote: post #67
rsynnott wrote: post #23

This idea that you can get good results from a bad process as long as you have good quality control seems… dubious, to say the least. “Sure, it’ll produce endless broken nonsense, but as long as someone is checking, it’s fine.” This, generally, doesn’t really work. You see people _try_ it in industry a bit; have a process which produces a high rate of failures, catch them in QA, rework (the US car industry used to be…

I'm not sure about the current state of the art, but microprocessors production is (was?) very bad. You make a lot of them in a single silicon wafer, and then test them thoughtfully until you find the few that are good. You drop all the defective ones because they are very cheap piece of sand and charge a lot for the ones that works correctly to cover all the costs. I'm not sure how this translates to programming, co…

[deleted]

Re: If you are good at code review, you will be good at using AI agents

#117

Post by dingnuts »

keeda wrote: post #115
rsynnott wrote: post #23

This idea that you can get good results from a bad process as long as you have good quality control seems… dubious, to say the least. “Sure, it’ll produce endless broken nonsense, but as long as someone is checking, it’s fine.” This, generally, doesn’t really work. You see people _try_ it in industry a bit; have a process which produces a high rate of failures, catch them in QA, rework (the US car industry used to be…

1. The flaw in this premise is that the process is bad. Aside from the countless anecdotal reports about how AI and agents are improving productivity, there are actual studies showing 25 - 55% boosts. Yes, RCTs at larger size than the METR one that keeps getting bandied about: https://news.ycombinator.com/item?id=44860577 and many more on Google Scholar: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&as_ylo...…

I picked one of the studies in the search (!) you linked. First of all, it's a bullshit debate tactic to try to overwhelm your opponents with vague studies -- a search is complete bullshit because it puts the onus on the other person to discredit the gargantuan amount of data you've flooded them with. Many of the studies in that search don't have anything to do with programming at all.

So right off the bat, I don't trust you. Anyway, I picked one study from the search to give you the benefit of the doubt. It compared leetcode in the browser to LLM generation. This tells us absolutely nothing about real world development.

What made the METR paper interesting was that they studied real projects, in the real world. We all know LLMs can solve well bounded problems in their data sets.

As for 3 I've seen a lot of broken nonsense. Let me know when someone vibe codes up a new mobile operating system or a competitor to KDE and Gnome lol

Re: If you are good at code review, you will be good at using AI agents

#118

Post by htrp »

Leynos wrote: post #84
rco8786 wrote: post #69

Earlier quoted context omitted.

> It’s very time consuming and 80% of the time I end up wondering if it would’ve been quicker to just do it all by myself right from the start. Yes, this. Every time I read these sort of step by step guides to getting the best results with coding agents it all just sounds like boatloads of work that erase the efficiency margins that AI is supposed to bring in the first place. And anecdotally, I've found that to be tr…

At least for me, I can have five of these processes running at once. I can also use Deepresearch for generating the designs with a survey of literature. I can use NotebookLM to analyse the designs. And I use Sourcery, CodeRabbit, Codex and Codescene together to do code review. It took me a long time to get there with custom cli tools and browser userscripts. The out of the box tooling is very limited unless you are w…

paid big bucks for devin... still was limited and not very good

Re: If you are good at code review, you will be good at using AI agents

#119

Post by z3c0 »

jvanderbot wrote: post #78
codingdave wrote: post #70

Earlier quoted context omitted.

Definitely sounds slower than doing it yourself. I am falling into a pattern of treating AI coding like a drunk mid-level dev: "I saw those few paragraphs of notes you wrote up on a napkin, and stayed up late Saturday night while drinking and spat out this implementation. you like?" So I can say to myself, "No, do not like. But the overall gist at least started in the right direction, so I can revise it from here and…

The most useful thing I've found is "I need to do X, show me 3 different popular libraries that do it". I've really limited my AI use to "Lady's Illustrated Primer" especially after some bad experiences with AI code from devs who should know better.

I don't even frame my requests conversationally. They usually read like brief demands, sometimes just comma delimited technologies followed by a goal. Works fine for me, but I also never prompt anything that I don't already understand how to do myself. Keeps the cart behind the horse.

Re: If you are good at code review, you will be good at using AI agents

#120

Post by Jensson »

thunky wrote: post #88
HarHarVeryFunny wrote: post #76

Earlier quoted context omitted.

Right, this is the exact opposite of the best practices that Edward Deming helped develop in Japan, then brought to the west. Quality needs to come from the process, not the people. Choosing to use a process known to be flawed, then hoping that people will catch the mistakes, doesn't seem like a great idea if the goal is quality. The trouble is that LLMs can be used in many ways, but only some of those ways play to t…

> Choosing to use a process known to be flawed, then hoping that people will catch the mistakes, doesn't seem like a great idea if the goal is quality. You're also describing the software development process prior to LLMs. Otherwise code reviews wouldn't exist.

People have built complex working mostly bug free products without code reviews so humans are not that flawed.

With humans and code reviews now two humans looked at it. With LLM and code review of the LLM output now one human looked at it, so its not the same. LLM are still far from as reliable as humans or you could just tell the LLM to do code reviews and then it builds the entire complex product itself.

Post reply on HN