[flagged]
My Agent Skill for Test-Driven Development
61–70 of 120 posts
Re: My Agent Skill for Test-Driven Development
#62[flagged]
Re: My Agent Skill for Test-Driven Development
#63One issue that I've run into with codex has been excessive use of fallbacks routines. Perhaps this is good practice in.professional programming in many situations, but for mine (in this case): computing geodesic distances and analysis, a silent bad fallback means the processed data is not what I thought it was..e.g. used an inaccurate geodesic method in place of the accurate one.
I HATE this. I call it speculative coding. Claude often calls it "defensive" programming. It's easily my #1 LLM pet peeve. I have yet to figure out a reliable way to make this stop happening.
I hate it as well. I have all sorts of skills and CLAUDE.md-based protections against it. I call it "a form of lying" to trigger ethics-related neurons, and I've also used linter rules and git pre-commit hooks to protect against this. I also don't ask for unit tests anymore, and instead ask for integration tests (with red/green TDD). I probably prevent 98% of the fallbacks/mocks with these methods, but some still slip through.
Re: My Agent Skill for Test-Driven Development
#64Tests are vanity in agentic engineering They do nothing to keep an AI on track in comparison to the aspects that simulate a product manager And the AI just will correct the test when it fails as opposed to correct the code, because the code didn't miss anything the specification changed My protip: just write tickets or have the AI write those too. that and the commits and the PRs will function as the AI’s memory bett…
My agent / skill files always tell it to trust neither the code or the test and to reason about the test failure which seems to work pretty well.
In another project without my rules I’ve noticed I have to tell it to set up data for playwright tests instead of skipping if none exists.
Re: My Agent Skill for Test-Driven Development
#65TDD sounds great on paper for agentic development but you quickly realize it balloons the token cost. Often I write some feature and then its repurposed or removed, code is refactored moved around as time goes. With TDD I would be taxed heavily and velocity slow to a crawl. The waterfall approach is better after trying out TDD especially when you have a multi-agent setup. Also I found that in some cases the tests wer…
> it balloons the token cost
how!!??
you write a test, which is one extra function. and maybe a paragraph or so per feature ("i made a RED test"... "i made it GREEN"), everything else is the same between normal development and TDD. this is chump change compared to the rest of development, including thinking tokens
Re: My Agent Skill for Test-Driven Development
#66This article would benefit from a date. It looks like it's recent (Internet Archive first grabbed it on May 29th) but it's the kind of information that can quickly become stale as models and agents improve. (I've been getting solid results recently from simply telling Claude Code and Codex "Test with uv run pytest, use red/green TDD".)
A lot of prompt engineering goes out of date quickly. Nobody nowadays goes "you are an expert software engineer. make no mistakes" lol. As a personal anecdote, I find that a lot of big prompts and skills use up context window budget and in many cases agents will eagerly try to use a skill even if it isn't super relevant or necessary for the current task. So when I have too many skills I have to spend a bunch of time…
I can't find the link now, but Anthropic has a post about using either a light model call or other logic (regex etc) to dynamically decide what tools to expose per incoming request.
I've run into the same issue and I still end up manually curtailing what's exposed to the model, limiting to the task at hand, but I like the idea of another (smaller I hope) model doing 70% of the clipping instead, automagically.
Re: My Agent Skill for Test-Driven Development
#67Earlier quoted context omitted.
I was a big proponent of encoding TDD red-green-refactor methodology into my agent workflows until recently when I made the same realization after reading this study: https://arxiv.org/pdf/2602.07900 TLDR; it found test-writing volume only weakly correlates with success and that encoding test-writing principles did not move resolution rates but _did_ materially change cost. Encouraging tests cost +19.8% output tokens…
Very interesting paper and it lines up exactly with my observations. The ROI just isn't there writing tests up front and the conclusion in that paper lays it out clearly Overall, these findings suggest that agent-written tests often behave more like a habitual software-development rou- tine than a dependable source of validation in this setting. More agent-written tests do not mean more solves; what they more reli- a…
Just chiming in to say that I've seen the exact same that you have. Tests are better used to help validate that was was generated worked after the fact.
That, and even the absolute SOTA models still suck at writing tests.
Which shouldn't be surprising: humans suck at it too most of the time...
Re: My Agent Skill for Test-Driven Development
#68Earlier quoted context omitted.
Very interesting paper and it lines up exactly with my observations. The ROI just isn't there writing tests up front and the conclusion in that paper lays it out clearly Overall, these findings suggest that agent-written tests often behave more like a habitual software-development rou- tine than a dependable source of validation in this setting. More agent-written tests do not mean more solves; what they more reli- a…
Just chiming in to say that I've seen the exact same that you have. Tests are better used to help validate that was was generated worked after the fact. That, and even the absolute SOTA models still suck at writing tests. Which shouldn't be surprising: humans suck at it too most of the time...
Absolutely, there's no reason to believe that agents will be more capable of writing tests than any other piece of code. The big pay off is actually verifying the code that was generated.
Re: My Agent Skill for Test-Driven Development
#69Re: My Agent Skill for Test-Driven Development
#70TDD sounds great on paper for agentic development but you quickly realize it balloons the token cost. Often I write some feature and then its repurposed or removed, code is refactored moved around as time goes. With TDD I would be taxed heavily and velocity slow to a crawl. The waterfall approach is better after trying out TDD especially when you have a multi-agent setup. Also I found that in some cases the tests wer…
I was a big proponent of encoding TDD red-green-refactor methodology into my agent workflows until recently when I made the same realization after reading this study: https://arxiv.org/pdf/2602.07900 TLDR; it found test-writing volume only weakly correlates with success and that encoding test-writing principles did not move resolution rates but _did_ materially change cost. Encouraging tests cost +19.8% output tokens…
The paper focuses on two things: default behavior and behavior with a prompt to write at least one new test.
In general — just like with humans — I find "just add more tests" to be counter-productive.
Tests make sense in a testable architecture: TDD can encourage one to be implicitly used, but it is a design, architectural choice that should be made explicit (lean to functional code; use direct, explicit dependency injection; ensure test stubs are just variants of the real implementation and fully tested using the same test as the real one...). LLMs should be prompted with this guidance instead for proper value estimation.