Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Toward automated verification of unreviewed AI-generated code

peterlavigne.com

Post reply on HN
Print view View original

71–80 of 100 posts

Re: Toward automated verification of unreviewed AI-generated code

#71

Post by raw_anon_1111 »

jryio wrote: post #8

This is a naïve approach, not just because it uses FizzBuzz, but because it ignores the fundamental complexity of software as a system of abstractions. Testing often involves understanding these abstractions and testing for/against them. For those of us with decades of experience and who use coding agents for hours per-day, we learned that even with extended context engineering these models are not magically covering…

He’s saying you should write or at least have the LLM write the tests and you carefully review the tests and not the code.

Re: Toward automated verification of unreviewed AI-generated code

#72

Post by mvrckhckr »

jryio wrote: post #8

This is a naïve approach, not just because it uses FizzBuzz, but because it ignores the fundamental complexity of software as a system of abstractions. Testing often involves understanding these abstractions and testing for/against them. For those of us with decades of experience and who use coding agents for hours per-day, we learned that even with extended context engineering these models are not magically covering…

The best way I can describe the approach I take is having the ability to "smell" what the AI might have gotten wrong (or forgotten completely).

It happens all the time, even when I only scan the code or simply run it and use it. It's uncanny how many such "smells" I find even with the most trivial applications. Sometimes its replies in Codex or Claude Code are enough to trigger it.

These are mistakes only a very (very) inexperienced developer would make.

Re: Toward automated verification of unreviewed AI-generated code

#73

Post by icoder »

Gigachad wrote: post #65
duskdozer wrote: post #20

So are we finally past the stage where people pretend they're actually reading any of the code their LLMs are dumping out?

I don't believe the junior devs on my team even run the code they are generating, let alone read it. Feeling like I'm doing 5x the work reviewing and testing that the person submitting has.

And I bet the impact of your review work on their development into medior/senior goes towards 0.

I recognize this discrepancy where review effort becomes more than the coding itself. I don't think I could sustain that for long.

Re: Toward automated verification of unreviewed AI-generated code

#76

Post by mattdeboard »

eggbrain wrote: post #68
mattdeboard wrote: post #62

Earlier quoted context omitted.

Do you not review junior developers' code? I don't understand your point

Your comment seems to imply AI is currently at a junior developer's level -- 12 months ago I would have agreed (like I mentioned in my parent comment, both near the end and about the "latter" team I was a part of), but it's gotten quite good over the past few months. When even Linus Torvalds compliments AI code (ref: https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Fa... ) I think we can say he wouldn't have…

Ok but are you arguing against code reviews of AI generated code?

Re: Toward automated verification of unreviewed AI-generated code

#77

Post by [deleted] »

Swizec wrote: post #45
dctoedt wrote: post #39

Earlier quoted context omitted.

> Let me guess, you've never worked in a real production environment? The comment to which you're responding includes a note at the end that the commenter is being sarcastic. Perhaps that wasn't in the comment when you responded to it.

It wasn’t thanks for highlighting. Can be hard to tell online because there’s a lot of people genuinely suggesting everyone should build their own software on the fly

[deleted]

Re: Toward automated verification of unreviewed AI-generated code

#78

Post by [deleted] »

Swizec wrote: post #18
loloquwowndueo wrote: post #12

Earlier quoted context omitted.

> is this going to let me grow the way I need it to in the future This doesn’t matter in the age of AI - when you get a new requirement just tell the AI to fulfill it and the old requirements (perhaps backed by a decent test suite?) and let it figure out the details, up to and including totally trashing the old implementation and creating an entirely new one from scratch that matches all the requirements. For perform…

> including totally trashing the old implementation and creating an entirely new one from scratch that matches all the requirements Let me guess, you've never worked in a real production environment? When your software supports 8, 9, 10 or more zeroes of revenue, "trash the old and create new" are just about the scariest words you can say. There's people relying on this code that you've never even heard of. Really go…

[deleted]

Re: Toward automated verification of unreviewed AI-generated code

#79

Post by slopinthebag »

baq wrote: post #54
hvb2 wrote: post #49

Earlier quoted context omitted.

So you're an auto maker, you say you can care about your product but not care how is built? If you're building for the cheapest segment of the market, just maybe. Anything else is a hard no imho

Yes? If you’re an auto factory, you might care, but an auto maker cares about minimizing cost and maximizing revenue within the regulatory constraints. Nowhere is there a requirement to care about how the car is built, there are requirements on what the car can and cannot do.

Caring about the requirements on what the car can and cannot do sounds suspiciously like caring about how it's built when you consider how it's built directly impacts what it can and cannot do.

Re: Toward automated verification of unreviewed AI-generated code

#80

Post by loloquwowndueo »

Swizec wrote: post #32
empath75 wrote: post #29

Earlier quoted context omitted.

> When your software supports 8, 9, 10 or more zeroes of revenue, "trash the old and create new" are just about the scariest words you can say. There's people relying on this code that you've never even heard of. Well, now it'll take them 5 minutes to rewrite their code to work around your change.

> Well, now it'll take them 5 minutes to rewrite their code to work around your change You misunderstand. It will take them 2 years to retrain 5000 people on the new process across hundreds of locations. In some fields, whole new college-level certifications courses will have to be created. In my specific experience it’s just a few dozen (maybe 100) people doing the manual process on top of our software and it takes…

> You misunderstand. It will take them 2 years to retrain 5000 people on the new process across hundreds of locations. In some fields, whole new college-level certifications courses will have to be created.

Replace them by AI.

I’m still being sarcastic.

Post reply on HN