Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How well do agents use test/verification techniques?

danluu.com

Post reply on HN
Print view View original

41–50 of 83 posts

Re: How well do agents use test/verification techniques?

#41

Post by MrJohz »

wesselbindt wrote: post #26
siscia wrote: post #5

It is still early, but I find that this experiment makes little to no sense and it is barely useful. The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself. 80%+ of effective testing is not in the testing framework but in the code architecture. The author doesn't mention how the code is being architected and managed. For what it is worth, I found that forcing agen…

> The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself. I'm not sure I fully understand. In my view, it's pretty evident that tests should test behavior and not structure. Coupling to implementation details is unavoidable but should be managed, and ideally minimized, to keep the test suite robust and maintainable. Given the choice between a test suite that monke…

The question "for which functions/modules/classes should I write tests?" and the question "how do I structure my functions/modules/classes?" are very closely related questions, which I think is what the other comment was getting at.

That follows largely from what you're saying though, so I think the two of you agree, you're just coming at the same thing from different perspectives. Like you say, you want to avoid/manage coupling tests to implementation details, and typically the best way to do that is to bundle a significant, testable chunk of code into a single module, define a clear public interface to that module, and then test that interface. The internals of that module are then free to change, but the test interface should stay the same.

But the corollary of that is that you can't just design your modules independently of your tests, you need to design them so that they are testable. Which is why the previous comment links testing to code architecture/design.

Regarding dependencies like that, the best situation is where you can either:

(a) design a module so that the dependency is injected with a clear interface (not just "here's an instance of libredis.rs" but "here's a series of callbacks for saving data, those callbacks might save to Redis, but they might be in-memory only, or they might save everything to the blockchain, either way it doesn't matter").

(b) just spin up an instance of Redis and test directly against that — it should be quick enough, and there's plenty of ways to ensure that the tests don't affect each other even while accessing a shared resource.

Re: How well do agents use test/verification techniques?

#43

Post by MrJohz »

sceptic123 wrote: post #36
kqr wrote: post #23

Earlier quoted context omitted.

I thought it pretty clear that the code was generated by the same agent that received the testing prompt, so there were no constraints on the code structure, and the testing strategy was known at the time the structure was generated.

But what's the initial setup? Was it greenfields every time? And _how_ were the different testing frameworks used? What's the AGENTS.md there? So many things can influence these tests in positive/negative ways that are not included in the write up or results. Seems like a lot of effort without much in the way of actual helpful detail.

The setup is described in the linked post about programming language vs efficiency/token cost: https://danluu.com/pl-tokens/#zstd

> For the first eval, I tried giving agents the zstd RFC (plus errata) and telling them to implement a complete zstd decoder (agents are stuck in a container without internet access). The tests were not given to agents. For something with the surface area of zstd, it's not really reasonable to expect that the tests cover every possible case. For example, even though zstd is a fairly well-tested piece of software, I once found a data corruption bug in zstd. The test suite isn't intended to find extreme corner cases that might be lurking for years and is instead intended to check various cases that can "easily" be derived from the RFC that should work.

So basically, the agents were given the zstd RFC, told to implement a decoder, and also told to use a particular testing strategy (or not, for the control runs).

Re: How well do agents use test/verification techniques?

#44

Post by coder-pm »

vetronauta wrote: post #4

If I understood correctly, the author had the agent perform manual mutation testing (write code, write passing tests, manually change the code to see some tests fail, then revert the change), rather than automated mutation testing. Why not use a mutation-testing framework and consider the build failed if a certain percentage of mutations are not killed? The article claims that the agents didn’t actually use TDD or mu…

Yes, automated and gated. Zero missed rather than percentage. This morning harness failed the agent written test that passed on the fixed code and also passed against the mutated code. Test looked fine, code reviews would approve it, only the gate caught it! The agent didn't game it. Why I said zero missed, not a percentage? Because only one mutation survived and percentage threshold would probably swallow it.

Does anyone else gate at zero rather than a percentage threshold?

Re: How well do agents use test/verification techniques?

#45

Post by a2ff6eeb0 »

quietraster wrote: post #20

nice setup for this. did any single technique clearly stand out, or did it mostly come down to the base model?

The answer is mostly "do nothing, the model will figure it out", with a side of "ask the model to check its work".

This matches my experience, where the job of the engineer is mostly copy pasting requirements, letting the model do the thinking, and then manually testing the results.

Re: How well do agents use test/verification techniques?

#46

Post by siscia »

wesselbindt wrote: post #26
siscia wrote: post #5

It is still early, but I find that this experiment makes little to no sense and it is barely useful. The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself. 80%+ of effective testing is not in the testing framework but in the code architecture. The author doesn't mention how the code is being architected and managed. For what it is worth, I found that forcing agen…

> The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself. I'm not sure I fully understand. In my view, it's pretty evident that tests should test behavior and not structure. Coupling to implementation details is unavoidable but should be managed, and ideally minimized, to keep the test suite robust and maintainable. Given the choice between a test suite that monke…

You are not wrong but if we go deeper we will find more nuances.

> tests should test behaviour and not structure.

Of what?

The whole idea of software architecture, and underlying of my messages, is to structure the code so that it is easy, but before easy, possible, to work at the right abstraction level.

Which allows to tests the behaviour of components and not their structure.

Trivial example, say your code read from a socket and manage the bytes with some CPU operations and write them to another socket. (You may recognise it is basically what a compressor do)

The way in which you manage read and write to the sockets will make dramatically simpler or much more difficult to write good tests.

If you adopt strategies like sans-io, you will see that the testing is almost trivial.

If all the logic sprawl up from the read syscall in a loop, you will notice how more challenging testing becomes.

---

To answer your question, the way I let LLMs write code is very DI (dependency injection) based.

A class never instantiates another class - all the dependencies are passed to the constructor. Including time. Including whatever DB iteraction.

The reason why I prefer this is that I can test each component at every level of abstraction. I don't have to. But I can.

My current approach is to force coverage higher than say 80% as default and then when a bug is discovered drill down to the components.

Re: How well do agents use test/verification techniques?

#47

Post by throwatdem12311 »

ivanzhaowy123 wrote: post #18

In my experience, agents often think of more edge cases than humans when writing unit tests. But under the guidance of certain skills, they can become mechanical and lose sight of the business logic. For example, when I use the Superpowers skill set, the agent proactively adopts TDD for every new feature. But its understanding of testing often stays superficial: if the user asks for a screen with a “Send” button, it…

This is not really my experience using TDD with Claude at all. When I tell it “a button should appear under x condition” it writes the test, and will often even verify the test by changing the code to fail the condition correctly before coming back and tell me it’s done.

Every once in a while one of those useless tests you mention will sneak in…but I still read the code and either just remove it or tell the agent to get rid of it. It happens so rarely it’s barely an inconvenience.

But I don’t use any skills or anything like that to drive it. I just have a line in my CLAUDE.md to follow TDD best practices.

Personally I find most skills like these “superpowers” are just bullshit and don’t really help at all.

Re: How well do agents use test/verification techniques?

#48

Post by yorwba »

sceptic123 wrote: post #36
kqr wrote: post #23

Earlier quoted context omitted.

I thought it pretty clear that the code was generated by the same agent that received the testing prompt, so there were no constraints on the code structure, and the testing strategy was known at the time the structure was generated.

But what's the initial setup? Was it greenfields every time? And _how_ were the different testing frameworks used? What's the AGENTS.md there? So many things can influence these tests in positive/negative ways that are not included in the write up or results. Seems like a lot of effort without much in the way of actual helpful detail.

The setup is described in an earlier post linked in the second paragraph. Agents are given the zstd RFC and told to implement it, greenfield. Testing frameworks were used however the agents decided, which is to say, as mentioned repeatedly throughout the post, badly or not at all. Other factors that could conceivably affect the results can be assumed to be held constant, since the goal is to see what happens when you vary the instructions regarding which testing methodology should be used.

Re: How well do agents use test/verification techniques?

#50

Post by simonwsimonw »

a2ff6eeb0 wrote: post #45
quietraster wrote: post #20

nice setup for this. did any single technique clearly stand out, or did it mostly come down to the base model?

The answer is mostly "do nothing, the model will figure it out", with a side of "ask the model to check its work". This matches my experience, where the job of the engineer is mostly copy pasting requirements, letting the model do the thinking, and then manually testing the results.

[dead]
Post reply on HN