I don't know what I'm most impressed by: (a) the testing expertise, (b) the time spent looking into the reasoning mistakes generated by LLMs, or (c) the insane amounts of money this must have cost!
How well do agents use test/verification techniques?
21–30 of 83 posts
Re: How well do agents use test/verification techniques?
#22nice setup for this. did any single technique clearly stand out, or did it mostly come down to the base model?
This is answered by the large plot early in TFA. All tests were made using the same model, and adding no special instructions at all provided performance above the average.
Re: How well do agents use test/verification techniques?
#23It is still early, but I find that this experiment makes little to no sense and it is barely useful. The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself. 80%+ of effective testing is not in the testing framework but in the code architecture. The author doesn't mention how the code is being architected and managed. For what it is worth, I found that forcing agen…
I thought it pretty clear that the code was generated by the same agent that received the testing prompt, so there were no constraints on the code structure, and the testing strategy was known at the time the structure was generated.
Re: How well do agents use test/verification techniques?
#24If I understood correctly, the author had the agent perform manual mutation testing (write code, write passing tests, manually change the code to see some tests fail, then revert the change), rather than automated mutation testing. Why not use a mutation-testing framework and consider the build failed if a certain percentage of mutations are not killed? The article claims that the agents didn’t actually use TDD or mu…
Would automated mutation testing actually make a big difference for agents, or would they just find new ways to game the metric?
I believe it would. Automated mutation testing is the test coverage metric that is nearly impossible to cheat.
Re: How well do agents use test/verification techniques?
#25If I understood correctly, the author had the agent perform manual mutation testing (write code, write passing tests, manually change the code to see some tests fail, then revert the change), rather than automated mutation testing. Why not use a mutation-testing framework and consider the build failed if a certain percentage of mutations are not killed? The article claims that the agents didn’t actually use TDD or mu…
This surprised me too. I suspect the robots only got a "use mutation testing" prompt and independently decided that it must mean manual mutation testing, or possibly that they ran in a sandbox where an automated mutation testing tool was not installed.
But I'm surprised Dan Luu didn't make a remark about this. Surely he must know the difference!
Re: How well do agents use test/verification techniques?
#26It is still early, but I find that this experiment makes little to no sense and it is barely useful. The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself. 80%+ of effective testing is not in the testing framework but in the code architecture. The author doesn't mention how the code is being architected and managed. For what it is worth, I found that forcing agen…
> The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself.
I'm not sure I fully understand. In my view, it's pretty evident that tests should test behavior and not structure. Coupling to implementation details is unavoidable but should be managed, and ideally minimized, to keep the test suite robust and maintainable.
Given the choice between a test suite that monkey patches out some dependency (redis, say) on a per-test basis, and one which substitutes a fake for this dependency in some centralized composition root, which one do you prefer, and why?
Re: How well do agents use test/verification techniques?
#27I wonder how well this fares try to make illegal states unrepresentable In my experience, agents know how to do it. They just don't if it's not the default style of the language.
Go is an amazing AI language for this reason.
The tests that get created also tend to be higher quality than say the slop I see in python.
Though I do suspect my codebases are doing heavy lifting in terms of steering towards quality outcomes.
Re: How well do agents use test/verification techniques?
#28[dead]
Re: How well do agents use test/verification techniques?
#29You know how everyone thinks agents are bad at the thing they're good at and good at the thing they're bad at? It turns out I must be bad at testing, because I thought they do a reasonable job.
Most developers are not good at testing. That's why the role of test engineer exists.
I've seen in multiple projects things like assertTrue(true).
I'm sure the agent is better in testing than average enterprise developer.
Re: How well do agents use test/verification techniques?
#30This is significantly more thorough than any testing I've done, and in a totally different domain, but my anecdotal experience getting agents to use Hypothesis was quite poor.
The agent really, really struggled to bridge the gap between the code and the actual business rules it was meant to be modelling. It also struggled to work out which functions at which layer were appropriate to write tests for. So, its tests tended to be very brittle to changes to domain logic.
Essentially, agents have always seemed to struggle with modularity and problem decomposition. Good testing is about finding the right things to test, which means working out how to subdivide the input state into an appropriate product state, and checking each behaviour independently. IMO, this is the most difficult and complicated thing about programming, so I won't say it struggled _more_ than a human would --- but humans have the advantage of being able to sleep on it?
One minor thing I observed was that it tended to get really hung up on floating point edge cases (NaNs, infinities). Maybe floating point edge cases are over-represented in the property-testing training data, but it's essentially irrelevant for my usecase, at least as far as the business rules go.