Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When AI writes the software, who verifies it?

leodemoura.github.io

Post reply on HN
Print view View original

131–140 of 309 posts

Re: When AI writes the software, who verifies it?

#131

Post by maltalex »

Maybe I'm missing something, but isn't this the same as writing code, but with extra steps?

Currently, engineers work with loose specifications, which they translate into code. With the proposed approach, they would need to first convert those specifications into a formally verifiable form before using LLMs to generate the implementation.

But to be production-ready, that spec would have to cover all possible use-cases, edge cases, error handling, performance targets, security and privacy controls, etc. That sounds awfully close to being an actual implementation, only in a different language.

Re: When AI writes the software, who verifies it?

#132

Post by throwaway613746 »

lateforwork wrote: post #121

The first thing you should have AI write is a comprehensive test suite. Then have it implement the main functionality. If the tests pass that is one level of verification. In addition you can have one AI check another AI's code. I routinely copy/paste code from Claude to ChatGPT and Gemini have them check each other's code. This works very well. During the process I have my own eyes verify the code as well.

[dead]

Re: When AI writes the software, who verifies it?

#133

Post by devin »

sjajzh wrote: post #107
throwaw12 wrote: post #99

Earlier quoted context omitted.

> Ideally, you’re working with teammates you trust. I do trust them, but code is not theirs, prompt is. What if I trust them, but because how much they use LLMs their brain started becoming lazy and they started missing edge cases, who should review the code? me or them? At the beginning, I relied on my trust and did quick scans, but eventually noticed they became un-interested in the craft and started submitting LLM…

Yeah, I agree with you. I’d say they’re not high performers anymore. Best answer I’ve got is find a place where quality matters. If you’re at a body shop it’s not gonna be fun. I do think some of this is just a hype wave and businesses will learn quality and trust matter. But maybe not - if wealth keeps becoming more concentrated at the top, it’s slop for the plebs.

My work has turned into churning out a PR, marking it as a draft so no one reviews it, and walking away. I come back after thinking about what it produced and usually realize it missed something or that the implications of some minor change are more far-reaching than the LLM understood. I take another pass. Then, I walk away again. Repeat.

Honestly I'm not sure much has changed with my output, because I don't submit PRs which aren't thoughtful. That is what the most annoying people in my organization do. They submit something that compiles, and then I spend a couple hours of my day demonstrating how incorrect it is.

For small fixes where I can recognize there is a clear, small fix which is easily testable I no longer add them to a TODO list, I simply set an agent off on the task and take it all the way to PR. It has been nice to be able to autopilot mindless changesets.

Re: When AI writes the software, who verifies it?

#134

Post by dehrmann »

fmbb wrote: post #78
madrox wrote: post #42

I encourage everyone to RTFA and not just respond to the headline. This really is a glimpse into where the future is going. I've been saying "the last job to be automated will be QA" and it feels more true every day. It's one thing to be a product engineer in this era. It's another to be working at the level the author is, where code needs to be verifiable. However, once people stop vibing apps and start vibing kerne…

There are still no successful useful vibe codes apps. Kernels are pretty far away I think.

Apps are a strange measure because there aren't really any new, groundbreaking ones. PCs and smartphones have mostly done what people have wanted them to do for a while.

Re: When AI writes the software, who verifies it?

#135

Post by heftykoo »

Another AI, obviously. And then a third AI to monitor the first two for conflicts of interest. Jokes aside, this is exactly the era where formal verification (like TLA+ or Lean, seeing the other post on the front page) actually makes commercial sense. If the code is generated, the only human output of value is the spec. We are moving from writing logic to writing constraints.

Re: When AI writes the software, who verifies it?

#136

Post by qsera »

bonoboTP wrote: post #100
fmbb wrote: post #78

Earlier quoted context omitted.

There are still no successful useful vibe codes apps. Kernels are pretty far away I think.

This is a very strange statement. People don't always announce when they use AI for writing their software since it's a controversial topic. And it's a sliding scale. I'm pretty sure a large fraction of new software has some AI involved in its development.

> new software has some AI involved in its development.

A large part of it is probably just using it as a better search. Like "How do I define a new data type in go?".

Re: When AI writes the software, who verifies it?

#137

Post by shimman »

dehrmann wrote: post #134
fmbb wrote: post #78

Earlier quoted context omitted.

There are still no successful useful vibe codes apps. Kernels are pretty far away I think.

Apps are a strange measure because there aren't really any new, groundbreaking ones. PCs and smartphones have mostly done what people have wanted them to do for a while.

There are plenty of ground breaking apps but they aren't making billions of advertising revenue, nor do they have large numbers. I honestly think torrent applications (and most peer to peer type of stuff) are very cool and very useful for small-medium groups but it'll never scale to a billion user thing.

Do agree it's a weird metric to have, but can't think of a better one outside of "business" but that still seems like a poor rubric because the vast majority of people care about things that aren't businesses and if this "life altering" technology basically amounts to creating digital slaves then maybe we as a species shouldn't explore the stars.

Re: When AI writes the software, who verifies it?

#138

Post by rvz »

redhed wrote: post #18
sausagefeet wrote: post #13

Earlier quoted context omitted.

I agree with you. But I have to say, it is an uphill battle and all the incentives are against you. 1. AI is meant to make us go faster, reviews are slow, the AI is smart, let it go. 2. There are plenty of AI maximizers who only think we should be writing design docs and letting the AI go to town on it. Maybe, this might be a great time to start a company. Maximize the benefits of AI while you can without someone who…

I actually agree with good time to start a company. Lot of available software engineers that can actually understand code, AI at a level that can actually speed up development, and so many startups focusing on AI wrapper slop that you can actually make a useful product and separate yourself from the herd. Or you can be a grifter and make some AI wrapper yourself and cash out with some VC investment. So good time for…

The people that are declaring holier-than-thou with self proclaimed 'principles' are the worst grifters and the ones actively scamming with AI with VC investment.

Pretending that they can only save the world and at the same time declaring they don't use AI but use it secretly by building an so-called "AI startup" and then going on the media doomsaying that "AGI" is coming.

At this point in this cycle in AI, "AGI" is just grifting until IPO.

Re: When AI writes the software, who verifies it?

#139

Post by roadbuster »

> The Claude C Compiler illustrates the other side: it optimizes for

> passing tests, not for correctness. It hard-codes values to satisfy

> the test suite. It will not generalize.

This is one of the pain points I am suffering at work: workers ask coding agents to generate some code, and then to generate test coverage for the code. The LLM happily churns out unit tests which are simply reinforcing the existing behaviour of the code. At no point does anyone stop and ask whether the generated code implements the desired functional behaviour for the system ("business logic").

The icing on the cake is that LLMs are producing so much code that humans are just rubber stamping all of it. Off to merge and build it goes.

I have no constructive recommendations; I feel the industry will keep their foot on the pedal until something catastrophic happens.

Re: When AI writes the software, who verifies it?

#140

Post by nvlled »

bfung wrote: post #118

When humans write the software, who verifies it? half sarcasm, half real-talk. TDD is nice, but human coders barely do it. At least AI can do it more!

> half sarcasm, half real-talk.

If you could pause a bit from being awed by your own perceived insightfulness, you would think a just bit harder and realize that LLMs can generate hundreds of thousands of code that no human could every verify within a finite amount of time. Human-written software is human verifiable, AI-assisted human-written software is still human verifiable to some extent, but purely AI-written software can no longer be verified by humans.

Post reply on HN