Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Agent Skill for Test-Driven Development

saturnci.com

Post reply on HN
Print view View original

111–120 of 120 posts

Re: My Agent Skill for Test-Driven Development

#111

Post by yaodub »

enraged_camel wrote: post #102
yaodub wrote: post #60

Earlier quoted context omitted.

Have you found integrating outputs from different frontier labs consistently improves final results, or is it just kind of voodoo?

It's useful, but increases review time and mental energy requirement. Often times Codex and Opus will find the same issues when given a review task, but will disagree on issue severity. Codex might claim that something is a blocker, while Opus will say it's just a medium/low. Or vice versa.

Same with legal questions, tbh. Spots the same issues, completely disagree on which ones matter. Maybe you just need a third model to choose between the outputs lol

Re: My Agent Skill for Test-Driven Development

#112

Post by esperent »

rsalus wrote: post #110
esperent wrote: post #88

Earlier quoted context omitted.

From that paper: > This raises a central question: do such tests meaningfully improve issue resolution, or do they mainly mimic a familiar software-development practice while consuming interaction budget? This is an important question but it's not the one I'm most interested in when requiring agents to follow TDD. My goal is to lock in behavior because it was happening way too frequently that an agent would successfu…

tests are an important signal of course, but the use case you describe doesn't necessarily mean you need to follow TDD. the data suggests that creating the tests after the code is just as or even more effective, and at significantly cheaper input cost. I've been finding enforcing integrations and behavior structurally (e.g., through codegen/schemagen, e2e tests, etc) more reliable than simply instructing the models t…

Why do you think creating tests later would be cheaper?

Re: My Agent Skill for Test-Driven Development

#114

Post by reg_dunlop »

zuzululu wrote: post #33
reg_dunlop wrote: post #27

Earlier quoted context omitted.

But that repurposing/removal is exactly what's avoided if you follow through with the SEF framework he outlines. I have to push back on the idea that token costs balloon when using TDD within the context of a strong framework such as Jason has laid out here. If the feature is repurposed/removed/refactored....I'd argue the specification wasn't well thought out prior to burning into tokens. We're so eager to do a lot o…

You cant spec out what you dont know, scope, requirements change from real world feedback

Then adjust the specs with the scope and feedback.

I fail to see the argument you're making...

Features aren't made in a vacuum. If specs are made/written....with the information available now...then it's better than not writing specs.

Writing specs with incomplete information is better than not writing specs with incomplete information.

Re: My Agent Skill for Test-Driven Development

#115

Post by [deleted] »

simonw wrote: post #8

This article would benefit from a date. It looks like it's recent (Internet Archive first grabbed it on May 29th) but it's the kind of information that can quickly become stale as models and agents improve. (I've been getting solid results recently from simply telling Claude Code and Codex "Test with uv run pytest, use red/green TDD".)

[deleted]

Re: My Agent Skill for Test-Driven Development

#116

Post by ArtRichards »

Nizoss wrote: post #105
krupan wrote: post #92

I find it hard to believe that these LLM systems with their enormous training sets and built-in system prompts have their output meaningfully modified by a few paragraphs of extra prompting in the form of these skill files, BUT, it is cool to see people writing out consise, focused documents like this. These would have great to have as a young developer, and great for several of the teams I've worked in in the past.…

I use a different approach, I enforce TDD using hooks. Think of it this way: You interact with your agent and ask it to implement a feature. Now every change it wants to make will have to be approved by a separate agent. This second agent is spawned using the SDK and can see the pending change, recent session history for context, instructions on how to interpret the information in relation to TDD, and any project cus…

Its all about retaining the context and spawning sub agents which can bootstrap quickly and accurately.

I'm interested in others dping something similar :) I included a docs cli tool in pypi to manage this context:

https://artrichards.github.io/agent-playbook-suite/blog/

Re: My Agent Skill for Test-Driven Development

#117

Post by victorbjorklund »

SubiculumCode wrote: post #49

One issue that I've run into with codex has been excessive use of fallbacks routines. Perhaps this is good practice in.professional programming in many situations, but for mine (in this case): computing geodesic distances and analysis, a silent bad fallback means the processed data is not what I thought it was..e.g. used an inaccurate geodesic method in place of the accurate one.

Yea, I have seen that too.

Re: My Agent Skill for Test-Driven Development

#118

Post by buster »

steno132 wrote: post #17

Test driven development is one of the worst ideas nowadays in the LLM age. We have models that can consistently write expert level, usually bug free code for you and rapidly fix even complex bugs in your codebase. The token cost and tech debt introduced by tests is just not worth it. There's usually no bugs and if there are, you can fix them quickly if and when it's needed.

No it's probably the most important idea.

Re: My Agent Skill for Test-Driven Development

#119

Post by bmitc »

Royce-CMR wrote: post #66
porphyra wrote: post #20

Earlier quoted context omitted.

A lot of prompt engineering goes out of date quickly. Nobody nowadays goes "you are an expert software engineer. make no mistakes" lol. As a personal anecdote, I find that a lot of big prompts and skills use up context window budget and in many cases agents will eagerly try to use a skill even if it isn't super relevant or necessary for the current task. So when I have too many skills I have to spend a bunch of time…

I can't find the link now, but Anthropic has a post about using either a light model call or other logic (regex etc) to dynamically decide what tools to expose per incoming request. I've run into the same issue and I still end up manually curtailing what's exposed to the model, limiting to the task at hand, but I like the idea of another (smaller I hope) model doing 70% of the clipping instead, automagically.

> Anthropic has a post about using either a light model call or other logic (regex etc) to dynamically decide what tools to expose per incoming request.

How? Using the agent SDK or Claude Code? If the latter, it'd be nice if they figured that out. There's a huge amount of quality of life things missing from Claude Code. It's a pretty raw frontend to the backend models. And either Claude Code or the backend models get convinced they don't need skills they've been asked to read or even built-in capabilities like reading PDFs.

Re: My Agent Skill for Test-Driven Development

#120

Post by bmitc »

SubiculumCode wrote: post #49

One issue that I've run into with codex has been excessive use of fallbacks routines. Perhaps this is good practice in.professional programming in many situations, but for mine (in this case): computing geodesic distances and analysis, a silent bad fallback means the processed data is not what I thought it was..e.g. used an inaccurate geodesic method in place of the accurate one.

> excessive use of fallbacks routines

What are "fallbacks routines"?

Post reply on HN