Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Agent Skill for Test-Driven Development

saturnci.com

Post reply on HN
Print view View original

91–100 of 120 posts

Re: My Agent Skill for Test-Driven Development

#91

Post by mathgeek »

oefrha wrote: post #87
porphyra wrote: post #20

Earlier quoted context omitted.

A lot of prompt engineering goes out of date quickly. Nobody nowadays goes "you are an expert software engineer. make no mistakes" lol. As a personal anecdote, I find that a lot of big prompts and skills use up context window budget and in many cases agents will eagerly try to use a skill even if it isn't super relevant or necessary for the current task. So when I have too many skills I have to spend a bunch of time…

> Nobody nowadays goes "you are an expert software engineer. make no mistakes" You know what, I checked Opus 4.8's instructions to a review subagent the other day and it literally opened with > You are a senior infrastructure/security engineer doing a thorough, adversarial code review... I didn't say anything like that myself.

Much like agents, I can tell myself I'm a senior infrastructure/security engineer doing a thorough, adversarial code review, but that doesn't change the results much.

Re: My Agent Skill for Test-Driven Development

#92

Post by krupan »

I find it hard to believe that these LLM systems with their enormous training sets and built-in system prompts have their output meaningfully modified by a few paragraphs of extra prompting in the form of these skill files, BUT, it is cool to see people writing out consise, focused documents like this. These would have great to have as a young developer, and great for several of the teams I've worked in in the past. I dabble with python for automating things here and there and I just learned some new things reading __mharison__'s skill in the comments here.

This kind of wisdom used to be cfound in blog posts, or in the beads of more senior developers, but they were never written out as concisely as these skill files. It's kinda funny that billions of dollars had to be spent creating a machine that's a rough human analog needing guidance to get us to produce these documents

Re: My Agent Skill for Test-Driven Development

#93

Post by jasonswett »

krupan wrote: post #92

I find it hard to believe that these LLM systems with their enormous training sets and built-in system prompts have their output meaningfully modified by a few paragraphs of extra prompting in the form of these skill files, BUT, it is cool to see people writing out consise, focused documents like this. These would have great to have as a young developer, and great for several of the teams I've worked in in the past.…

The reason it works is because there's a difference between the model knowing something and the agent doing something. Claude will happily write giant untested functions even though it "knows" that short functions are easier to understand and then testing enables safe refactoring etc. The model also "knows" many conflicting "facts", such as the fact that testing is smart and that testing is a waste of time. It can't act on both beliefs at the same time. That's why nudging it toward your own preferred behaviors works.

Re: My Agent Skill for Test-Driven Development

#94

Post by turlockmike »

jasonswett wrote: post #93
krupan wrote: post #92

I find it hard to believe that these LLM systems with their enormous training sets and built-in system prompts have their output meaningfully modified by a few paragraphs of extra prompting in the form of these skill files, BUT, it is cool to see people writing out consise, focused documents like this. These would have great to have as a young developer, and great for several of the teams I've worked in in the past.…

The reason it works is because there's a difference between the model knowing something and the agent doing something. Claude will happily write giant untested functions even though it "knows" that short functions are easier to understand and then testing enables safe refactoring etc. The model also "knows" many conflicting "facts", such as the fact that testing is smart and that testing is a waste of time. It can't…

It lacks a critical self, but the weights are there for in context learning and nudging behaviour. It's goal is to complete whatever task is given. You need to make sure the outcome you want is clearly defined.

You don't need elaborate prompts, just a few lines

"All code must have corresponding tests written ahead of time to prove the code meets the specification" is sufficient for most use cases. Prose can help nudge it more if it isn't adhearing consistently.

Re: My Agent Skill for Test-Driven Development

#96

Post by __mharrison__ »

necovek wrote: post #71
__mharrison__ wrote: post #28

Testing is so important for development. Even more so when coding with agents. I think it is the probably the biggest lever to keep AI in guardrails. (It's also why I wrote my latest book, Effective Testing, because I routinely find that my clients are very poor at treating.)

Having thought heavily and even presenting on exactly the same topics, looking at the ToC, your book seems to cover the basics well. However, since we are talking about effectiveness, applying a lot of these principles might lead to a non-maintainable codebase — for humans and LLMs alike. When any change causes 500 tests to break, or it causes nothing to break (see monkey-patching and/or mocking), you've gotten to a…

My experience with refactoring is if a change causes larger number of tests faults, I run the last failed test and fix that (see my AGENTS.md posted elsewhere). Generally, if you fix the one issue everything else falls in line.

Wrt mocking. I'm not a huge fan. Again, look at my AGENTS.md. I prefer monkeypatch as a last resort option. Luckily, if you use TDD, you rarely have to use mocking. If you don't use TDD...

Re: My Agent Skill for Test-Driven Development

#97

Post by gruez »

jasonswett wrote: post #93
krupan wrote: post #92

I find it hard to believe that these LLM systems with their enormous training sets and built-in system prompts have their output meaningfully modified by a few paragraphs of extra prompting in the form of these skill files, BUT, it is cool to see people writing out consise, focused documents like this. These would have great to have as a young developer, and great for several of the teams I've worked in in the past.…

The reason it works is because there's a difference between the model knowing something and the agent doing something. Claude will happily write giant untested functions even though it "knows" that short functions are easier to understand and then testing enables safe refactoring etc. The model also "knows" many conflicting "facts", such as the fact that testing is smart and that testing is a waste of time. It can't…

Isn't all of what you described what post-training/RLHF is supposed to do? The internet is full of racism, so if you're just predicting the next token based on training data, you'll get racism (eg. Microsoft Tay), but that's more or less solved by AI companies now.

Re: My Agent Skill for Test-Driven Development

#98

Post by vikramkr »

krupan wrote: post #92

I find it hard to believe that these LLM systems with their enormous training sets and built-in system prompts have their output meaningfully modified by a few paragraphs of extra prompting in the form of these skill files, BUT, it is cool to see people writing out consise, focused documents like this. These would have great to have as a young developer, and great for several of the teams I've worked in in the past.…

Why is that hard to believe? It's literally the prompt telling it what to do - if you want a poem about watermelons you tell it to write a poem about watermelons, if you want tests you tell it to write tests. It's not like TDD is some universal pattern that every llm will naturally optimize towards

Re: My Agent Skill for Test-Driven Development

#99

Post by vikramkr »

gruez wrote: post #97
jasonswett wrote: post #93

Earlier quoted context omitted.

The reason it works is because there's a difference between the model knowing something and the agent doing something. Claude will happily write giant untested functions even though it "knows" that short functions are easier to understand and then testing enables safe refactoring etc. The model also "knows" many conflicting "facts", such as the fact that testing is smart and that testing is a waste of time. It can't…

Isn't all of what you described what post-training/RLHF is supposed to do? The internet is full of racism, so if you're just predicting the next token based on training data, you'll get racism (eg. Microsoft Tay), but that's more or less solved by AI companies now.

The post training is meant to make it more steerable (usually). I might not want it to write tests. I might not have a dev environment set up for an agent to run tests in it's loop. A major goal in post training is to make it follow instructions, which doesn't have to mean have particular instincts for code organization

Re: My Agent Skill for Test-Driven Development

#100

Post by tclancy »

mathgeek wrote: post #91
oefrha wrote: post #87

Earlier quoted context omitted.

> Nobody nowadays goes "you are an expert software engineer. make no mistakes" You know what, I checked Opus 4.8's instructions to a review subagent the other day and it literally opened with > You are a senior infrastructure/security engineer doing a thorough, adversarial code review... I didn't say anything like that myself.

Much like agents, I can tell myself I'm a senior infrastructure/security engineer doing a thorough, adversarial code review, but that doesn't change the results much.

You have to be looking in a mirror and slap your face a couple of times to make it work.
Post reply on HN