Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How well do agents use test/verification techniques?

danluu.com

Post reply on HN
Print view View original

31–40 of 83 posts

Re: How well do agents use test/verification techniques?

#31

Post by jiaosdjf »

ivanzhaowy123 wrote: post #18

In my experience, agents often think of more edge cases than humans when writing unit tests. But under the guidance of certain skills, they can become mechanical and lose sight of the business logic. For example, when I use the Superpowers skill set, the agent proactively adopts TDD for every new feature. But its understanding of testing often stays superficial: if the user asks for a screen with a “Send” button, it…

Probably because thats how most companies do it because most corporate workers are lazy box-tickers who are long fed up with the processes. It really feels like we're forcing human processes onto AI.

What actually is the point of TDD? - If its to force you to think about edge cases early before you've started building the feature then that sounds like a human trait - If its to be living documentation then that sounds like a human trait

We're going into weird rabbit holes where we've mismatched the tool that is AI which produces extremely cheap code very quickly - with the processes that we've built for slow and expensive to write human-generated code.

Re: How well do agents use test/verification techniques?

#33

Post by crabbone »

I haven't finished reading and had to make a detour into another article by the same author, so, probably, I will have to extend the comment, but so far this:

> Perhaps the limiting factor is just that knowledge of effective test techniques isn't very widespread

That linked to: https://danluu.com/testing/

But of course... the problem of testing is a lot harder than performance optimization... I'm surprised this comes as a surprise. Performance optimization has plenty of evaluation metrics by its very nature. Testing? -- I wish there was anything tangible at all... Because we have metrics for optimization, we have theories of optimization, i.e. we have a way of explaining how or what optimization should do. With testing? -- we are nowhere close to this point.

Another aspect of this disparity is that we also know how to sell performance optimizations. It's easy to write into an ad pamphlet that the version 2.0 of gobbledygook does 185% more gobbledygook than the 1.0! (The number faithfully copied from my cereal box!) With testing? -- How can you even tell the customer that the product was tested better? Swear on your life and cross your heart (twice, as opposed to the last time when you only did it once?)

In general, in the field, I've only have so far met with extreme pessimism about feasibility of "theory of testing" existing. Even though the need for testing goes without saying, the actual testing task is reserved for the least competent and there's little no no effort made to improve anything in this department as it's perceived to be a black hole in the budget: no matter how much you could spend on testing, the effect is likely to be the same.

Re: How well do agents use test/verification techniques?

#34

Post by zaphirplane »

ponector wrote: post #29
anitil wrote: post #8

You know how everyone thinks agents are bad at the thing they're good at and good at the thing they're bad at? It turns out I must be bad at testing, because I thought they do a reasonable job.

Most developers are not good at testing. That's why the role of test engineer exists. I've seen in multiple projects things like assertTrue(true). I'm sure the agent is better in testing than average enterprise developer.

big tech is moving / has moved away from the role.

Re: How well do agents use test/verification techniques?

#36

Post by sceptic123 »

kqr wrote: post #23
siscia wrote: post #5

It is still early, but I find that this experiment makes little to no sense and it is barely useful. The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself. 80%+ of effective testing is not in the testing framework but in the code architecture. The author doesn't mention how the code is being architected and managed. For what it is worth, I found that forcing agen…

I thought it pretty clear that the code was generated by the same agent that received the testing prompt, so there were no constraints on the code structure, and the testing strategy was known at the time the structure was generated.

But what's the initial setup? Was it greenfields every time? And _how_ were the different testing frameworks used? What's the AGENTS.md there? So many things can influence these tests in positive/negative ways that are not included in the write up or results. Seems like a lot of effort without much in the way of actual helpful detail.

Re: How well do agents use test/verification techniques?

#39

Post by salmonellaeater »

ckvibubueu wrote: post #27
ngruhn wrote: post #12

I wonder how well this fares try to make illegal states unrepresentable In my experience, agents know how to do it. They just don't if it's not the default style of the language.

Go is an amazing AI language for this reason. The tests that get created also tend to be higher quality than say the slop I see in python. Though I do suspect my codebases are doing heavy lifting in terms of steering towards quality outcomes.

I've found the opposite. Go doesn't even have sum types, so it's hard to use constructive data modeling techniques to model the domain. The only tool available for ensuring exhaustive handling of all cases is interfaces (Visitor pattern) which is verbose.
Post reply on HN