Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do agents.md files help coding agents?

twitter.com

Post reply on HN
Print view View original

41–50 of 52 posts

Re: Do agents.md files help coding agents?

#41

Post by ch4s3 »

hnlmorg wrote: post #38
ch4s3 wrote: post #36

Earlier quoted context omitted.

Yeah I wrote my own language and obviously it’s not in any training data which end tip being an interesting experiment with agents. I’ve found that a few concise skills and an agents.md make a huge difference in guiding the LLMs. Specifically getting them to use the all in one build tool, which the LLMs won’t use without direction.

As a language author myself, I'd be interested to learn more about how you utilised agents.md and skills for this language of yours.

The core design decision that supports everything is having a sort of multi-tool that does a lot for the language called forge. It builds, tests, runs, initializes, formats, manages dpes, and so on. It also has a search tool that apparently works kind of like Haskell's hoogle(which I discovered later), and that search tool helps agents find code.

I have some usage instructions about this tool in my various agents files which works well enough alongside a syntax.md that can be copied around.

From there I have some concise skills under /.claude/commands/ like build-test, forge-search, and a few things for working on the compiler. The specific skills include short snippets, descriptions, and some concise usage guidelines.

It all works reasonably well.

My biggest issue is that as I build out libraries I keep finding edge cases with my Perceus GC[1] implementation.

[1] https://www.microsoft.com/en-us/research/wp-content/uploads/...

Re: Do agents.md files help coding agents?

#42

Post by avadodin »

dofm wrote: post #34
popcorncowboy wrote: post #30

Agents.md is just a prompt pre-pend. This is like asking "do prompts help coding agents".

But that is an open question. For example some smaller, local-friendly LLMs (Gemma, Qwen) clearly do better without a long system prompt. Faster ad just no worse at code. Both seem eminently capable of writing high quality, eloquent code without being specifically prompted to, both need little instruction on tool calling. The consensus is leading us towards very long system prompts, agent files, stuffing with skills,…

You can just ask any model to go do a code review and it will probably do better than no review at all but with rules you can make sure at least it processed inputs enough to produce a hallucinated reason which you can check for why a rule doesn't apply.

To me, that's worth losing some theoretical model performance and token efficiency unless you're using the small local models which basically lose all of that while trying to follow the rules.

Re: Do agents.md files help coding agents?

#43

Post by RugnirViking »

sceptic123 wrote: post #40
RugnirViking wrote: post #10

yes, they do. I think people overindex on this paper, I remember when it came out we had a lot of discussion in my company about it. But its clear to see they do at least change the agent's behavior, and things like telling it "always use xyz version of java, use gradle to build the project, use this command to run the tests" are really important instead of letting it fumble about trying to find the right thing every…

That is all the stuff that should be in the README.md of the project in the first place though.

right! I don't disagree. README and agents.md probably will end up looking similar (or being the same) in the long run - readmes should probably have MORE information about the structure of the code if anything

Re: Do agents.md files help coding agents?

#44

Post by rsfern »

dofm wrote: post #35
wiseowise wrote: post #5

You putting “you’re an expert jerk off master” in agents.md is the same as shaman burning a bone to predict a future.

I am equally cynical! Some of it is cosmic ordering, some of it is cargo culting, some of it is clearly FUD. The trick presumably is to end up on the words that actively shift the “culture” of the subsequent interpretation of words. I don’t know how you do that, really, but I do think it’s extremely instructive to see what a model can do with an empty prompt.

If the explicit role-playing prompt is just to identify multi-valent terms, then revising the question to include more specific context without a role-play prompt should work just as well right? I’d be really interested if anyone has evaluated that hypothesis

A fun (frustrating) feature of language is that we get these name collisions even with a single domain. One that I have to remember to revise myself fairly often these days when chatting with other experts in my field is “diffusion model” which can either mean generative deep learning or a differential equation describing mass transport.

Re: Do agents.md files help coding agents?

#45

Post by swingboy »

sceptic123 wrote: post #40
RugnirViking wrote: post #10

yes, they do. I think people overindex on this paper, I remember when it came out we had a lot of discussion in my company about it. But its clear to see they do at least change the agent's behavior, and things like telling it "always use xyz version of java, use gradle to build the project, use this command to run the tests" are really important instead of letting it fumble about trying to find the right thing every…

That is all the stuff that should be in the README.md of the project in the first place though.

Yes, but harnesses don’t automatically include the README.md in the system prompt like they do AGENTS.md.

Re: Do agents.md files help coding agents?

#46

Post by dofm »

rsfern wrote: post #44
dofm wrote: post #35

Earlier quoted context omitted.

I am equally cynical! Some of it is cosmic ordering, some of it is cargo culting, some of it is clearly FUD. The trick presumably is to end up on the words that actively shift the “culture” of the subsequent interpretation of words. I don’t know how you do that, really, but I do think it’s extremely instructive to see what a model can do with an empty prompt.

If the explicit role-playing prompt is just to identify multi-valent terms, then revising the question to include more specific context without a role-play prompt should work just as well right? I’d be really interested if anyone has evaluated that hypothesis A fun (frustrating) feature of language is that we get these name collisions even with a single domain. One that I have to remember to revise myself fairly ofte…

LoRA/LoRa always makes me chuckle. Feels like a turf war. And in the photography world, "AI model" is overloaded.

And indeed — beyond specific guidance around tool management, you have to wonder if a very minimal prompt and a /grill-me or /ubiquitous-language approach, narrowed to focus on what matters this time, wouldn't do better. I have been playing with this idea as I try to learn all this stuff in a local model context.

Re: Do agents.md files help coding agents?

#48

Post by dofm »

avadodin wrote: post #42
dofm wrote: post #34

Earlier quoted context omitted.

But that is an open question. For example some smaller, local-friendly LLMs (Gemma, Qwen) clearly do better without a long system prompt. Faster ad just no worse at code. Both seem eminently capable of writing high quality, eloquent code without being specifically prompted to, both need little instruction on tool calling. The consensus is leading us towards very long system prompts, agent files, stuffing with skills,…

You can just ask any model to go do a code review and it will probably do better than no review at all but with rules you can make sure at least it processed inputs enough to produce a hallucinated reason which you can check for why a rule doesn't apply. To me, that's worth losing some theoretical model performance and token efficiency unless you're using the small local models which basically lose all of that while…

Definitely. Though my own experimentation is showing that with the small models it is worth working through the fewest words. I don't think Gemma will need too much prompting to not sound like a web dev blogger.

Re: Do agents.md files help coding agents?

#49

Post by tstrimple »

prodigycorp wrote: post #18
deaux wrote: post #17

[dupe] https://news.ycombinator.com/item?id=47034087 Paper was discussed here 4 months ago, and the linked tweet on this post doesn't add any insights and completely misses the huge caveats that come with the found result: the main benefits of using AGENTS.md files are inherently opposed to the characteristics of _median_ "public github project that has an AGENTS.md file".

Yeah it was a good discussion. I'm definitely on the side of "a well written agents.md is very good for the agent". The file should not be static. The agents.md from a year ago is not the right agents.md for today. Many of them are overly long, overly instructive, and include a lot of unccessary code bits. I think it's useful for people using agent harness to regularly evaluate your skills, agent rules, and memory im…

I use CC for a lot of things outside of generating code. One of my CC contexts is for converting recipes to dairy/gluten free by recommending dish appropriate replacements. It very clearly has a huge influence on it to the point where I really don't understand how the counterpoint could even be argued.

Re: Do agents.md files help coding agents?

#50

Post by lousclues »

SeriousM wrote: post #33

Yes. In my case (and I guess everyones usecase is subjective) my system prompt states to read the AGENT.md file when possible. On a new project I usually set up the context of the model (language to use, reason of the product/prototype, etc.) and then I tell the LLM to write a AGENT.md, STATE.md and ROADMAP.md. I don't tell the LLM what's in there because the model has it's own directive and flavor what should be in…

I do something similar. I don’t necessarily call it AGENT or STATE and every project has its own files. I have architecture documents that accompany change log descriptions that load technical knowledge that the agent can readily use.

I find it also necessary to have a principles document outlining the particular problems that the software is supposed to solve and guard rails to not cross. I call it promise driven development.

Post reply on HN