Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Agent Skill for Test-Driven Development

saturnci.com

Post reply on HN
Print view View original

71–80 of 120 posts

Re: My Agent Skill for Test-Driven Development

#71

Post by necovek »

__mharrison__ wrote: post #28

Testing is so important for development. Even more so when coding with agents. I think it is the probably the biggest lever to keep AI in guardrails. (It's also why I wrote my latest book, Effective Testing, because I routinely find that my clients are very poor at treating.)

Having thought heavily and even presenting on exactly the same topics, looking at the ToC, your book seems to cover the basics well.

However, since we are talking about effectiveness, applying a lot of these principles might lead to a non-maintainable codebase — for humans and LLMs alike.

When any change causes 500 tests to break, or it causes nothing to break (see monkey-patching and/or mocking), you've gotten to a point where your testing approach is ineffective.

Most start applying principles of just enough tests and testable architectures too late, yet I believe they are fundamental.

Do you cover these in your book?

Re: My Agent Skill for Test-Driven Development

#73

Post by csbartus »

This specify-encode-fulfill loop/method is effective to make agents create bug-free code.

In my version of this workflow I do specify myself, then let the LLM do the rest.

This way 1.) I'm 100% sure the understanding/spec is good 2.) It's translated into an executable format so the implementation can be verified 3.) The implementation has maximum code coverage tests which steers the AI to produce code which follows standards, fits into the existing codebase, and it's very easy to refactor.

So far, this is the one and only advantage of using LLMs in my SWE practice. They glue together (human written) specs with code, with confidence, in no time.

Re: My Agent Skill for Test-Driven Development

#75

Post by dchuk »

fowlie wrote: post #36

Haven't tried this, but I've recently become a big fan of Matt Pococks skills. Workflow: /grill-with-docs -> /to-prd -> /to-issue -> /tdd. That will interview relentlessy until there is a "shared understanding" using "ubiquitous language", then it will spec all requirements with user stories, create issues and implement them using tdd.

Been using his skills a lot lately, they are wonderful. I’ve added an issue to specs skill that grounds the issues with a technical plan against the current codebase, and a research school that spawns a bunch of agents to look up best practices on the internet for those issues with specs, it really dials things in. I need to issue a PR to his project for those two…

Re: My Agent Skill for Test-Driven Development

#76

Post by pramodbiligiri »

zuzululu wrote: post #23

TDD sounds great on paper for agentic development but you quickly realize it balloons the token cost. Often I write some feature and then its repurposed or removed, code is refactored moved around as time goes. With TDD I would be taxed heavily and velocity slow to a crawl. The waterfall approach is better after trying out TDD especially when you have a multi-agent setup. Also I found that in some cases the tests wer…

My approach (with LLMs especially) aligns more with what's outlined in "Growing OO Software Guided by Tests" (https://growing-object-oriented-software.com/toc.html). Chapter 4 there says "First, Test a Walking Skeleton", and Chapter 5 has "Start Each Feature with an Acceptance Test". I think it comes down to: get something working end-to-end first in a verifiable way, and then keep refining both the feature and its tests (preferable with TDD).

I've noticed that LLMs tend to generate multiple testcases in one shot (which is not how humans usually go about TDD), and also they don't start with Integration Tests, unless instructed to do so.

Re: My Agent Skill for Test-Driven Development

#77

Post by Rohunyyy »

fowlie wrote: post #36

Haven't tried this, but I've recently become a big fan of Matt Pococks skills. Workflow: /grill-with-docs -> /to-prd -> /to-issue -> /tdd. That will interview relentlessy until there is a "shared understanding" using "ubiquitous language", then it will spec all requirements with user stories, create issues and implement them using tdd.

It seems like he got the skills from BMAD method https://docs.bmad-method.org/tutorials/getting-started/

Re: My Agent Skill for Test-Driven Development

#78

Post by bob1029 »

TDD is fundamentally problematic in every practical implementation I've ever seen. I don't think the same thing, but much faster, is going to help at all. TDD tends to cause adverse, higher order effects.

I am currently observing AI authored tests creating a massive sense of complacency because a human no longer owns responsibility for the test suite. It's too easy to reject ownership by way of the various agent prompting schemes. I find myself enjoying the idea of it too, primarily because adding tests to even the most trivial functionality is mandatory due to the TDD policy.

Developing good tests is like an artform. Total coverage is a terrible objective. Correctness does not compose upward. It's a game of chasing ghosts if you think you can build a perfectly clean system bottom up and then magically meet the customer at the top. They're gonna kick your jenga tower over on day one.

Re: My Agent Skill for Test-Driven Development

#79

Post by emigre »

bfeynman wrote: post #59
rsalus wrote: post #46

Earlier quoted context omitted.

not necessarily, TDD has little bearing on output quality

In what world or frame of reference would doing TDD have "little" bearing on output quality? If you build a system around satisfying some set of requirements it seems logical that output quality would have pretty heavy correlation.

It's possible to satisfy a set of requirements with code that's low quality. There's the maintainability of the code, for example, or the performance of the system.

Re: My Agent Skill for Test-Driven Development

#80

Post by kirtivr »

fowlie wrote: post #36

Haven't tried this, but I've recently become a big fan of Matt Pococks skills. Workflow: /grill-with-docs -> /to-prd -> /to-issue -> /tdd. That will interview relentlessy until there is a "shared understanding" using "ubiquitous language", then it will spec all requirements with user stories, create issues and implement them using tdd.

TIL. Thanks for that!
Post reply on HN