When AI writes the software, who verifies it? • Hacker News Board

Re: When AI writes the software, who verifies it?

#171

Post by gwern » Wed, Mar 04, 2026, 4:29 AM UTC

madrox wrote: post #42

I encourage everyone to RTFA and not just respond to the headline. This really is a glimpse into where the future is going. I've been saying "the last job to be automated will be QA" and it feels more true every day. It's one thing to be a product engineer in this era. It's another to be working at the level the author is, where code needs to be verifiable. However, once people stop vibing apps and start vibing kerne…

> I encourage everyone to RTFA and not just respond to the headline.

This is an example of an article which 'buries the lede'†.

It should have started with the announcement of the new zlib autoformalization (!) https://leodemoura.github.io/blog/2026/02/28/when-ai-writes-... to get you excited.

Then it should have talked about the rest - instead of starting with rather graceless and ugly LLM-written generic prose about AI topics that to many readers is already tiresomely familiar and doubtless was tldr for even the readers who aren't repelled automatically by that.

† or in my terms, fails to 'make you care': https://gwern.net/blog/2026/make-me-care

Re: When AI writes the software, who verifies it?

#172

Post by SoftTalker » Wed, Mar 04, 2026, 4:30 AM UTC

roadbuster wrote: post #139

> The Claude C Compiler illustrates the other side: it optimizes for > passing tests, not for correctness. It hard-codes values to satisfy > the test suite. It will not generalize. This is one of the pain points I am suffering at work: workers ask coding agents to generate some code, and then to generate test coverage for the code. The LLM happily churns out unit tests which are simply reinforcing the existing behavi…

Um, you're supposed to write the tests first. The agents can't do this?

Re: When AI writes the software, who verifies it?

#173

Post by tbossanova » Wed, Mar 04, 2026, 4:31 AM UTC

MartyMcBot wrote: post #160

roadbuster wrote: post #139

> The Claude C Compiler illustrates the other side: it optimizes for > passing tests, not for correctness. It hard-codes values to satisfy > the test suite. It will not generalize. This is one of the pain points I am suffering at work: workers ask coding agents to generate some code, and then to generate test coverage for the code. The LLM happily churns out unit tests which are simply reinforcing the existing behavi…

[flagged]

Once upon a time people advocated writing tests first…

Re: When AI writes the software, who verifies it?

#174

Post by tbossanova » Wed, Mar 04, 2026, 4:31 AM UTC

g947o wrote: post #164

ojo-rojo wrote: post #157

Earlier quoted context omitted.

How about a subsequent review where a separate agent analyzes the original issue and resultant code and approves it if the code meets the intent of the issue. The principle being to keep an eye out for manual work that you can describe well enough to offload. Depending on your success rate with agents, you can have one that validates multiple criteria or separate agents for different review criteria.

You are fighting nondeterministic behavior with more nondeterministic behavior, or in other words, fighting probability with probability. That doesn't necessarily make things any better.

Probably true

(Sorry.)

Re: When AI writes the software, who verifies it?

#175

Post by daliusd » Wed, Mar 04, 2026, 4:38 AM UTC

SoftTalker wrote: post #172

roadbuster wrote: post #139

> The Claude C Compiler illustrates the other side: it optimizes for > passing tests, not for correctness. It hard-codes values to satisfy > the test suite. It will not generalize. This is one of the pain points I am suffering at work: workers ask coding agents to generate some code, and then to generate test coverage for the code. The LLM happily churns out unit tests which are simply reinforcing the existing behavi…

Um, you're supposed to write the tests first. The agents can't do this?

They can, but should be explicitly told to do that. Otherwise they just everything in batches. Anyway pure TDD or not but tests catches only what you tell AI to write. AI does not now what is right, it does what you told it to do. The above problem wouldn’t be solved by pure TDD.

Re: When AI writes the software, who verifies it?

#176

Post by pyridines » Wed, Mar 04, 2026, 4:44 AM UTC

g947o wrote: post #164

ojo-rojo wrote: post #157

Earlier quoted context omitted.

How about a subsequent review where a separate agent analyzes the original issue and resultant code and approves it if the code meets the intent of the issue. The principle being to keep an eye out for manual work that you can describe well enough to offload. Depending on your success rate with agents, you can have one that validates multiple criteria or separate agents for different review criteria.

You are fighting nondeterministic behavior with more nondeterministic behavior, or in other words, fighting probability with probability. That doesn't necessarily make things any better.

In my experience, an agent with "fresh eyes", i.e., without the context of being told what to write and writing it, does have a different perspective and is able to be more critical. Chatbots tend to take the entire previous conversational history as a sort of canonical truth, so removing it seems to get rid of any bias the agent has towards the decisions that were made while writing the code.

I know I'm psychologizing the agent. I can't explain it in a different way.

Re: When AI writes the software, who verifies it?

#177

Post by krackers » Wed, Mar 04, 2026, 4:46 AM UTC

Spivak wrote: post #169

neya wrote: post #161

The fundamental problem is the verification loop for the average developer is grounded not in tests, but with results . Write code, reload browser, check output. Does it work the way I want? Good. We're done here. Not write code, write tests, ensure all test-cases are covered. Now, imagine such a flaky foundation is used to build on top of even more untested code. That's how bad quality software (that's usually unfix…

My impression has been with frontend tests is they come in two flavors: extremely hard to write to the point of impracticality, and useless. Most orgs settle for the latter and end up testing easily testable things like the final rendered dom and rely on human qa for all the hard bits like "does the page actually look like it's supposed to, does interacting with the UI elements behave correctly, and do the flows actu…

>can't meaningfully see and interact with the page like the end user will

Isn't this a great use case for LLM tests? Have a "computer use agent" and then describe the parameters of the test as "load the page, then navigate to bar, expect foo to happen". You don't need the LLM to generate a test using puppeteer or whatever which is coupled to the specific dom, you just describe what should happen.

Re: When AI writes the software, who verifies it?

#178

Post by codegladiator » Wed, Mar 04, 2026, 4:51 AM UTC

tbossanova wrote: post #173

MartyMcBot wrote: post #160

Earlier quoted context omitted.

[flagged]

Once upon a time people advocated writing tests first…

once upon a time 'engineering' in software had some meaning attached to it...

no other engineering profession would accept the standards(or rather their lack of) on which software engineering is running.

Re: When AI writes the software, who verifies it?

#179

Post by phantomathkg » Wed, Mar 04, 2026, 4:52 AM UTC

Human writes the requirements, contains flaw. Human or AI translate that to specifications, and eventually code.

It does not matter if the middle man is human or AI, or written in "traditional language" or "formal verification". Bugs will be there as human failed to defined a bullet proof requirements.

Re: When AI writes the software, who verifies it?

#180

Post by WhyNotHugo » Wed, Mar 04, 2026, 4:53 AM UTC

roadbuster wrote: post #139

> The Claude C Compiler illustrates the other side: it optimizes for > passing tests, not for correctness. It hard-codes values to satisfy > the test suite. It will not generalize. This is one of the pain points I am suffering at work: workers ask coding agents to generate some code, and then to generate test coverage for the code. The LLM happily churns out unit tests which are simply reinforcing the existing behavi…

This is why you write the tests first and then the code. Especially when fixing bugs, since you can be sure that the test properly fails when the bug is present.