[flagged]
How well do agents use test/verification techniques?
11–20 of 82 posts
Re: How well do agents use test/verification techniques?
#12I wonder how well this fares
try to make illegal states unrepresentable
Re: How well do agents use test/verification techniques?
#13You know how everyone thinks agents are bad at the thing they're good at and good at the thing they're bad at? It turns out I must be bad at testing, because I thought they do a reasonable job.
I thought so too but it's probably because I never review the tests. I made an exception recently and found tons of
assert(CONSTANT_CONFIG == valueOfConfig)
assert(prompt.includes("repo url"))Re: How well do agents use test/verification techniques?
#14You know how everyone thinks agents are bad at the thing they're good at and good at the thing they're bad at? It turns out I must be bad at testing, because I thought they do a reasonable job.
I don't think these results show that.
I think they show that additional prompting for testing approaches have wide differences in error rate (worst has twice the error rate of the best) but actually no extra prompting is pretty fine and most custom prompts are worse than no prompt.
Re: How well do agents use test/verification techniques?
#15If I understood correctly, the author had the agent perform manual mutation testing (write code, write passing tests, manually change the code to see some tests fail, then revert the change), rather than automated mutation testing. Why not use a mutation-testing framework and consider the build failed if a certain percentage of mutations are not killed? The article claims that the agents didn’t actually use TDD or mu…
Would automated mutation testing actually make a big difference for agents, or would they just find new ways to game the metric?
Re: How well do agents use test/verification techniques?
#16[flagged]
Re: How well do agents use test/verification techniques?
#17[flagged]
Re: How well do agents use test/verification techniques?
#18In my experience, agents often think of more edge cases than humans when writing unit tests. But under the guidance of certain skills, they can become mechanical and lose sight of the business logic.
For example, when I use the Superpowers skill set, the agent proactively adopts TDD for every new feature. But its understanding of testing often stays superficial: if the user asks for a screen with a “Send” button, it first writes a test checking whether the button exists. The test fails, so it adds the button to make it pass.
As a result, the test suite fills up with low-value cases that check whether a property exists or a string matches exactly. The agent follows the “write a failing test, then implement the feature” workflow, but never really tests the business behavior: When should sending be allowed? What should happen after success or failure? How should duplicate submissions be handled?
The problem isn’t that agents can’t write tests. It’s that they seem prone to reducing TDD to a rigid sequence of steps, struggling to independently derive meaningful test cases from business requirements and use them to drive development.
Re: How well do agents use test/verification techniques?
#19[flagged]
Re: How well do agents use test/verification techniques?
#20nice setup for this. did any single technique clearly stand out, or did it mostly come down to the base model?