Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How well do agents use test/verification techniques?

danluu.com

Post reply on HN
Print view View original

81–90 of 90 posts

Re: How well do agents use test/verification techniques?

#81

Post by Havoc »

I've had good luck with audit technique, specifically with mixing LLMs that have very different heritage.

Mostly write code with GLM5.3 and found that using Muse 1.3 to make detailed summary bugs works well, especially for Rust. Somehow muse is really good at rust. And from what I can tell even using the original model (GLM here) against that list to go fix it seems fine even if it is the model that made the mistake.

They seem to write 100s of tests too, but don't really have much confidence in those tbh much like author. I view it as a bonus. Tests are computationally cheap so knock yourself out Mr AI.

Re: How well do agents use test/verification techniques?

#84

Post by ckvibubueu »

salmonellaeater wrote: post #39
ckvibubueu wrote: post #27

Earlier quoted context omitted.

Go is an amazing AI language for this reason. The tests that get created also tend to be higher quality than say the slop I see in python. Though I do suspect my codebases are doing heavy lifting in terms of steering towards quality outcomes.

I've found the opposite. Go doesn't even have sum types, so it's hard to use constructive data modeling techniques to model the domain. The only tool available for ensuring exhaustive handling of all cases is interfaces (Visitor pattern) which is verbose.

Data modelling is helpful for humans that can expand that kind of context on the fly and with recall. For the midwit models I use it just creates context that is expressed once and then glossed over.

Having to over express the data model everywhere is precisely what makes AI good at using it.

Does the code suck to read? Sure, but I don't really do that very often anymore anyway.

In this case, the verbosity of go is a feature not a bug.

I guess it depends what your optimizing for, good code or fast turn around without needing to constantly remind the AI about certain type behaviors.

Re: How well do agents use test/verification techniques?

#85

Post by sceptic123 »

yorwba wrote: post #73
sceptic123 wrote: post #71

Earlier quoted context omitted.

So basically "one-shot using testing methodology". I don't think that's a very effective test of an agents ability.

It's intended as a "test to see how effective agents are when guided by someone with no expertise in testing who's maybe heard that you should apply certain techniques or use certain libraries." What do you think would be a more effective test for that purpose?

So not "How well do agents use test/verification techniques?" like the title says then.

It just looks like a lot of effort to go to in a way that doesn't answer the question the title poses.

I'd love to see something that could guide how to set an agent up for success rather than this naive approach which only really answers the questions that most of the audience here probably don't have.

Re: How well do agents use test/verification techniques?

#86

Post by genxy »

gregwebs wrote: post #74
jakevoytko wrote: post #64

Earlier quoted context omitted.

As someone who also spent a bunch of time benchmarking various techniques, this is as good as you can do without publishing a formal versioned benchmark suite that you want to maintain and run forever at immense cost to yourself. If you actually go to benchmark your own flow as you mentioned, you will quickly run into like a dozen problems that discourage you from publishing. - Are you sure that temperature and other…

I am really grateful that he’s publishing as well. I am just asking for the bare minimum to actually understand what has been tested and for reproducibility of methods- the publication of the prompts. It would take a lot less time than all the guess work analysis write up and be a lot more useful. Without the prompts the rest of your questions about reproducibility are moot.

This post boils down to, "I tried a bunch of things that didn't work very well". But we don't actually know what he tried. Everything hinges on Dan's prompting skills.

Re: How well do agents use test/verification techniques?

#87

Post by genxy »

ngruhn wrote: post #13
anitil wrote: post #8

You know how everyone thinks agents are bad at the thing they're good at and good at the thing they're bad at? It turns out I must be bad at testing, because I thought they do a reasonable job.

I thought so too but it's probably because I never review the tests. I made an exception recently and found tons of assert(CONSTANT_CONFIG == valueOfConfig) or tests for keywords in prompts: assert(prompt.includes("repo url"))

Was your prompt, "make lazy ineffectual tests?" Tests are code as well, they take the same effort to create, if not more. You need to specify the properties you want to test.

Re: How well do agents use test/verification techniques?

#89

Post by vetronauta »

coder-pm wrote: post #44
vetronauta wrote: post #4

If I understood correctly, the author had the agent perform manual mutation testing (write code, write passing tests, manually change the code to see some tests fail, then revert the change), rather than automated mutation testing. Why not use a mutation-testing framework and consider the build failed if a certain percentage of mutations are not killed? The article claims that the agents didn’t actually use TDD or mu…

Yes, automated and gated. Zero missed rather than percentage. This morning harness failed the agent written test that passed on the fixed code and also passed against the mutated code. Test looked fine, code reviews would approve it, only the gate caught it! The agent didn't game it. Why I said zero missed, not a percentage? Because only one mutation survived and percentage threshold would probably swallow it. Does a…

Do you have specific strategies on handling equivalent mutants?

Re: How well do agents use test/verification techniques?

#90

Post by vetronauta »

andai wrote: post #53
ivanzhaowy123 wrote: post #18

In my experience, agents often think of more edge cases than humans when writing unit tests. But under the guidance of certain skills, they can become mechanical and lose sight of the business logic. For example, when I use the Superpowers skill set, the agent proactively adopts TDD for every new feature. But its understanding of testing often stays superficial: if the user asks for a screen with a “Send” button, it…

I asked Sonnet to port a prolog interpreter from Python to JS. I was surprised and delighted that it ended up writing a test suite for it. All tests passed. Until... I inspected the code: it was just a bunch of print statements that said "test passed!" and didn't actually test anything. That was last year, so hopefully it doesn't do that anymore.

Few months ago I was debugging an integration with a no-code library that ships code in a custom binary format. I still smile thinking at the following test I read

assertThrows(RuntimeException.class, () -> callFoo(notABinaryFile)); //well, foo library is very complex, obviously this test throws!

Post reply on HN