Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When AI writes the software, who verifies it?

leodemoura.github.io

Post reply on HN
Print view View original

201–210 of 309 posts

Re: When AI writes the software, who verifies it?

#201

Post by Foobar8568 »

8note wrote: post #188
roadbuster wrote: post #139

> The Claude C Compiler illustrates the other side: it optimizes for > passing tests, not for correctness. It hard-codes values to satisfy > the test suite. It will not generalize. This is one of the pain points I am suffering at work: workers ask coding agents to generate some code, and then to generate test coverage for the code. The LLM happily churns out unit tests which are simply reinforcing the existing behavi…

> At no point does anyone stop and ask whether the generated code implements the desired functional behaviour for the system ("business logic"). its fun having LLMs because it makes it quite clear that a lot of testing has been cargo-culting. did people ever check often that the tests check for anything meaningful?

15years ago, I had tester writing "UI tests" / "User tests" that matched what the software was cranking out. At that time I just joined to continue at the client side so I didn't really worked on anything yet.

I had a fun discussion when the client tried to change values... Why is it still 0? Didn't you test?

And that was at that time I had to dive into the code base and cry.

Re: When AI writes the software, who verifies it?

#202

Post by cagz »

> No one is formally verifying the result

This might be the case for a hobby project or a start-up MVP being created in a rush, but in reality, there are a few points we may want to take into account:

1. Software teams I work with are maintaining the usual review practices. Even if a feature is completely created by AI. It goes through the usual PR review process. The dev may choose "Accept All", although I am not saying this is a good practice, the change still gets reviewed by a human.

2. From my experience, sub-agents intended for code and security review do a good job. It is even possible to use another model to review the code, which can provide a different perspective.

3. A year ago, code written by AI was failing to run the first time, requiring a painful joint troubleshooting effort. Now it works 95% of the time, but perhaps it is not optimal. Given the speed at which it is improving, it is safe to expect that in 6-9 months' time, it will not only work but will also be written to a good quality.

Re: When AI writes the software, who verifies it?

#203

Post by Foobar8568 »

pyridines wrote: post #176
g947o wrote: post #164

Earlier quoted context omitted.

You are fighting nondeterministic behavior with more nondeterministic behavior, or in other words, fighting probability with probability. That doesn't necessarily make things any better.

In my experience, an agent with "fresh eyes", i.e., without the context of being told what to write and writing it, does have a different perspective and is able to be more critical. Chatbots tend to take the entire previous conversational history as a sort of canonical truth, so removing it seems to get rid of any bias the agent has towards the decisions that were made while writing the code. I know I'm psychologizi…

Fresh eyes, some contexts and another LLM.

The problem is information fatigue from all the agents+code itself.

Re: When AI writes the software, who verifies it?

#205

Post by byzantinegene »

usefulcat wrote: post #194
WhyNotHugo wrote: post #180

Earlier quoted context omitted.

This is why you write the tests first and then the code. Especially when fixing bugs, since you can be sure that the test properly fails when the bug is present.

Agreed 1000%. But that can be a lot of work; creating a good set of tests is nearly as much or often even more effort than implementing the thing being tested. When LLMs can assist with writing useful tests before having seen any implementation, then I’ll be properly impressed.

from experience, AI is bad at TDD. they can infer tests based on written code, but are bad at writing generalised test unless a clear requirement is given, so you the engineer is doing most of the work anyway.

Re: When AI writes the software, who verifies it?

#206

Post by kazinator »

> What would a verification platform for the AI era require? A small, trusted kernel: a few thousand lines of code that check every step of every proof mechanically.

The code is a stochastic parrot job produced with zero algorithmic understanding.

Out of what magic unicorn's ass are you going to get a matching proof for it, to feed to this trusted kernel?

Re: When AI writes the software, who verifies it?

#207

Post by DeathArrow »

roadbuster wrote: post #139

> The Claude C Compiler illustrates the other side: it optimizes for > passing tests, not for correctness. It hard-codes values to satisfy > the test suite. It will not generalize. This is one of the pain points I am suffering at work: workers ask coding agents to generate some code, and then to generate test coverage for the code. The LLM happily churns out unit tests which are simply reinforcing the existing behavi…

>LLM happily churns out unit tests which are simply reinforcing the existing behaviour of the code. At no point does anyone stop and ask whether the generated code implements the desired functional behaviour for the system ("business logic").

You can use spec driven development and TDD. Write the tests first. Write failing code. Modify the code to pass the tests.

Re: When AI writes the software, who verifies it?

#209

Post by globular-toast »

I'm starting to think of this LLM thing a bit like fossil fuels.

We've got fossil fuels that were deposited over millions of years, a timescale we are not even properly equipped to imagine. We've been tapping that reserve for a few decades and it's caused all kinds of problems. We've painted ourselves into a corner and can't get out.

Now we've got a few decades worth of software to tap. When you use an LLM you don't create anything new, you just recycle what's already there. How long until we find ourselves in a very similar corner?

The inability of people to think ahead really astounds me. Sustainability should be at the forefront of everyone's mind, but it's barely even an afterthought. Rather, people see a tap running and just drink from it without questioning once where the water is coming from. It's a real animal brain thing. It'll get you as far as reproducing, but that's about it.

Post reply on HN