The written specs and the acceptance tests.
When AI writes the software, who verifies it?
261–270 of 309 posts
Re: When AI writes the software, who verifies it?
#262Earlier quoted context omitted.
At my job we have a requirement for 100% test coverage. So everyone just uses AI to generate 10,000 line files of unit tests and nobody can verify anything.
Exactly! It's frustrating how much developers get blamed for the outcomes of incompetent management.
> everyone just uses AI to generate 10,000 line files of unit tests and nobody can verify anything
This is not a guaranteed outcome of requiring 100% coverage. Not that that's a good requirement, but responding badly to a bad requirement is just as bad.
Re: When AI writes the software, who verifies it?
#263Earlier quoted context omitted.
once upon a time 'engineering' in software had some meaning attached to it... no other engineering profession would accept the standards(or rather their lack of) on which software engineering is running.
> no other engineering profession would accept the standards(or rather their lack of) on which software engineering is running. I have bad news for you: they are pushing those "standards" (Agile, ASPICE) also in hardware and mechanical engineering. The results can be seen already. Testing is expensive and this is the field where most savings can be implemented.
Agile isn't a coding standard or approach.
Re: When AI writes the software, who verifies it?
#264Earlier quoted context omitted.
I do that all the time with Claude. What part is not working?
I don't really use anthropic models. But when I tried it with others they can write tests but they never confirm that they fail before they proceed to make implementation that causes them to pass. Maybe I didn't prompt it forcibly enough.
I haven’t tried this (yet), but I’ve heard of people disabling write access to test code while the agent is writing implementation and vice versa. I imagine “disabling” could be done via prompting, or just a quick one liner like: chmod -r 0644 ./tests
Re: When AI writes the software, who verifies it?
#265Users
Re: When AI writes the software, who verifies it?
#266Earlier quoted context omitted.
But it's so BORING. AI gets to do the fun part (writing code) and I'm stuck with the lame bits. It's like watching someone else solve a puzzle, or watching someone else play a game vs playing it yourself (at least that's half as interesting as playing it through)
For me the most fun part is getting something that works. Design the goal, but not micromanage and get lost in the details. I love AI for that, but it is hard really owning code this way. (At least I manually approve every or most changes, but still, verifying is hard).
> For me the most fun part is getting something that works. Design the goal, but not micromanage and get lost in the details.
That’s management, which is not necessarily bad by itself.
What is bad if you have to be both a manager giving the requirements and the IC responsible for the output. That’s the worst of both worlds.
If it was possible to truly vibe code in the same way a product manager asks a team to build something then we wouldn’t need to have this discussion.
In reality you can never truly trust the output. Ultimately you’re on the hook if production breaks apart, not the LLM.
Re: When AI writes the software, who verifies it?
#267Earlier quoted context omitted.
It’s because people will do what they’re incentivized to do. And if no one cares about anything but whether the next feature goes out the door, that’s what programmers will focus on. Honestly I think the other thing that is happening is that a lot of people who know better are keeping their mouths shut and waiting for things to blow up. We’re at the very peak of the hype cycle right now, so it’s very hard to push bac…
Or if you say we should slow down your competence is questioned by others who are going very fast (and likely making mistakes we won't find until later). And there is an element of uncertainty. Am I just bad at using these new tools? To some degree probably, but does that mean I'm totally wrong and we should be going this fast?
There is a saying: slow is smooth and smooth is fast.
I have personally outpaced some of my more impatient colleagues by spending extra time up front setting up test harnesses, reading specifications, etcetera. When done judiciously it pays off in time scales of weeks or less.
Re: When AI writes the software, who verifies it?
#268> The Claude C Compiler illustrates the other side: it optimizes for > passing tests, not for correctness. It hard-codes values to satisfy > the test suite. It will not generalize. This is one of the pain points I am suffering at work: workers ask coding agents to generate some code, and then to generate test coverage for the code. The LLM happily churns out unit tests which are simply reinforcing the existing behavi…
This is why you write the tests first and then the code. Especially when fixing bugs, since you can be sure that the test properly fails when the bug is present.
I'd argue the AI writing the tests shouldn't even know about the implementation at all. You only want to pass it the interface (or function signatures) together with javadocs/docstrings/equivalent.
Re: When AI writes the software, who verifies it?
#269The first thing you should have AI write is a comprehensive test suite. Then have it implement the main functionality. If the tests pass that is one level of verification. In addition you can have one AI check another AI's code. I routinely copy/paste code from Claude to ChatGPT and Gemini have them check each other's code. This works very well. During the process I have my own eyes verify the code as well.
I was anticipating that having AI write code to pass tests (human and/or AI written tests) would be worthwhile, but in practice, I've found that even models such as Opus 4.6 Thinking, High Effort simply "cheats", or rather, fails to generalize much too often. It's occurred to me that perhaps I need some amount of randomness in the tests to keep the models honest, but it feels wrong. We'll see.