Snake oil. Just ask the model, all these custom agents/skills haven't proven that useful in practice.
Ya, if im constantly asking a model to do TDD development, you know what would make it a lot easier? A skill.
My Agent Skill for Test-Driven Development
11–20 of 120 posts
Re: My Agent Skill for Test-Driven Development
#12This article would benefit from a date. It looks like it's recent (Internet Archive first grabbed it on May 29th) but it's the kind of information that can quickly become stale as models and agents improve. (I've been getting solid results recently from simply telling Claude Code and Codex "Test with uv run pytest, use red/green TDD".)
Me too, although I dislike the fact that it over-focuses on mocks (which I accept is over-represented in the training data).
Re: My Agent Skill for Test-Driven Development
#13I believe using a skill here is the wrong approach. LLMs already know what TDD is and how to do it, just like object oriented programming.
If this is encoded in a skill, that skill essentially has to be loaded for everything thing your LLM is doing. This is probably one of the few areas where direct instructions via AGENTS.md is best, and I don't believe it requires much direction here to force the issue.
But I think the OP is just trying to have their agent work in a very specific way -- that is fine too.
> 5. Show me the test and ask for approval before continuing
Re: My Agent Skill for Test-Driven Development
#14Snake oil. Just ask the model, all these custom agents/skills haven't proven that useful in practice.
Nah. Skills are great. But you should write your own.
Re: My Agent Skill for Test-Driven Development
#15Snake oil. Just ask the model, all these custom agents/skills haven't proven that useful in practice.
not that i know much about the effectiveness of these skill files, i find it odd to call something given for free "snake oil", which i thought referred to the sale of fraudulent products (to the benefit of the snake oil salesperson), typically around healthcare-related stuff.
i think gp is calling skills snakeoil in genral
Re: My Agent Skill for Test-Driven Development
#16Earlier quoted context omitted.
Skills already are "just asking the model". Unless you'd prefer to type out the same instructions every single time? Skills are literally just Markdown documents that get loaded into context when the /skill-name is invoked.
i belive gp means llms produce what they see in training data/rl there isnt much too much customization you can do with skills. they are being sold as more powerful than they are. Like llms are intelligent blank slates that can be customized with mere markdown files.
I don't understand this line of criticism exactly. By putting new information in the context window, you are materially changing the activations at your point of sampling, which is literally "customizing with mere markdown files."
Taken to the extreme, the attitude that there is some special incantation that will unlock all capabilities is silly, and a lot of the "prompt engineering" discourse is similarly kind of dumb, but in-context learning is clearly a real thing.
Re: My Agent Skill for Test-Driven Development
#17Test driven development is one of the worst ideas nowadays in the LLM age. We have models that can consistently write expert level, usually bug free code for you and rapidly fix even complex bugs in your codebase.
The token cost and tech debt introduced by tests is just not worth it. There's usually no bugs and if there are, you can fix them quickly if and when it's needed.
Re: My Agent Skill for Test-Driven Development
#18Earlier quoted context omitted.
i belive gp means llms produce what they see in training data/rl there isnt much too much customization you can do with skills. they are being sold as more powerful than they are. Like llms are intelligent blank slates that can be customized with mere markdown files.
I don't understand this line of criticism exactly. By putting new information in the context window, you are materially changing the activations at your point of sampling, which is literally "customizing with mere markdown files." Taken to the extreme, the attitude that there is some special incantation that will unlock all capabilities is silly, and a lot of the "prompt engineering" discourse is similarly kind of du…
even if that works one time you can never be sure that your customization is in place or fell out of context's important zone or is contradicted by later context . you've reverted back to base llm behavior.
you are treating skill like sure thing
Re: My Agent Skill for Test-Driven Development
#19This article would benefit from a date. It looks like it's recent (Internet Archive first grabbed it on May 29th) but it's the kind of information that can quickly become stale as models and agents improve. (I've been getting solid results recently from simply telling Claude Code and Codex "Test with uv run pytest, use red/green TDD".)
Me too, although I dislike the fact that it over-focuses on mocks (which I accept is over-represented in the training data).
sometimes I also feel it tries to optimise for "per line coverage" over more "real, complex use cases" type tests
Re: My Agent Skill for Test-Driven Development
#20This article would benefit from a date. It looks like it's recent (Internet Archive first grabbed it on May 29th) but it's the kind of information that can quickly become stale as models and agents improve. (I've been getting solid results recently from simply telling Claude Code and Codex "Test with uv run pytest, use red/green TDD".)
A lot of prompt engineering goes out of date quickly. Nobody nowadays goes "you are an expert software engineer. make no mistakes" lol.
As a personal anecdote, I find that a lot of big prompts and skills use up context window budget and in many cases agents will eagerly try to use a skill even if it isn't super relevant or necessary for the current task. So when I have too many skills I have to spend a bunch of time toggling the checkboxes to figure out which ones are needed for the task at hand before starting...