Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do agents.md files help coding agents?

twitter.com

Post reply on HN
Print view View original

31–40 of 52 posts

Re: Do agents.md files help coding agents?

#31

Post by nikcub »

prodigycorp wrote: post #18
deaux wrote: post #17

[dupe] https://news.ycombinator.com/item?id=47034087 Paper was discussed here 4 months ago, and the linked tweet on this post doesn't add any insights and completely misses the huge caveats that come with the found result: the main benefits of using AGENTS.md files are inherently opposed to the characteristics of _median_ "public github project that has an AGENTS.md file".

Yeah it was a good discussion. I'm definitely on the side of "a well written agents.md is very good for the agent". The file should not be static. The agents.md from a year ago is not the right agents.md for today. Many of them are overly long, overly instructive, and include a lot of unccessary code bits. I think it's useful for people using agent harness to regularly evaluate your skills, agent rules, and memory im…

> "a well written agents.md is very good for the agent"

while even a mildly bad agents.md can be _very_ bad for the agent. they rot very quickly which is why human curation is essential.

same with memory - a lot of the self-learning tools that are becoming popular now degrade agents over time - which is why you end up being able to run an eval with no context and it performs better

> but that's why agentic coding can still be considered a "skill".

yes - far too many cases of throwing a kitchen sink of prompts, skills, tools etc. thinking the llm will sort it out. you need to constantly prune, eval, tweak, observe, update etc. in a loop

Re: Do agents.md files help coding agents?

#32

Post by iLoveOncall »

nicolailolansen wrote: post #22
RugnirViking wrote: post #10

yes, they do. I think people overindex on this paper, I remember when it came out we had a lot of discussion in my company about it. But its clear to see they do at least change the agent's behavior, and things like telling it "always use xyz version of java, use gradle to build the project, use this command to run the tests" are really important instead of letting it fumble about trying to find the right thing every…

Yeah. It's very easy to give a definitive "yes", if you've ever worked with coding agents in any capacity. I use them in much the same way as you, there are a bunch of things that would be nice for the agent to know, specifically for me, the project, or how I would like it to run as an example: git or coding review loops ( i use roborev ). Asking the agent to do these things every time, is very time consuming.

Maybe read the paper... It says that the only thing that are helpful is indeed what you describe here: basic commands and context about running / working on the project, rather than information about the business or technological aspects of the project itself.

Re: Do agents.md files help coding agents?

#33

Post by SeriousM »

Yes. In my case (and I guess everyones usecase is subjective) my system prompt states to read the AGENT.md file when possible.

On a new project I usually set up the context of the model (language to use, reason of the product/prototype, etc.) and then I tell the LLM to write a AGENT.md, STATE.md and ROADMAP.md. I don't tell the LLM what's in there because the model has it's own directive and flavor what should be in these files. The models already know the purpose of these files by themself! On a new session, I let the agent read the markdown files in order to continue with the work. Before a session ends, I let the LLM update the markdown files. Maybe one word of caution: don't switch models - it's like putting another person on a working station and ask them to continue the work of others.

Easy setup, really good outcome!

Re: Do agents.md files help coding agents?

#34

Post by dofm »

popcorncowboy wrote: post #30

Agents.md is just a prompt pre-pend. This is like asking "do prompts help coding agents".

But that is an open question. For example some smaller, local-friendly LLMs (Gemma, Qwen) clearly do better without a long system prompt. Faster ad just no worse at code. Both seem eminently capable of writing high quality, eloquent code without being specifically prompted to, both need little instruction on tool calling. The consensus is leading us towards very long system prompts, agent files, stuffing with skills, when it might be cargo-culting (again).

A culture of really long preambles certainly benefits cloud AI companies though!

Re: Do agents.md files help coding agents?

#35

Post by dofm »

wiseowise wrote: post #5

You putting “you’re an expert jerk off master” in agents.md is the same as shaman burning a bone to predict a future.

I am equally cynical!

Some of it is cosmic ordering, some of it is cargo culting, some of it is clearly FUD.

The trick presumably is to end up on the words that actively shift the “culture” of the subsequent interpretation of words.

I don’t know how you do that, really, but I do think it’s extremely instructive to see what a model can do with an empty prompt.

Re: Do agents.md files help coding agents?

#36

Post by ch4s3 »

RugnirViking wrote: post #10

yes, they do. I think people overindex on this paper, I remember when it came out we had a lot of discussion in my company about it. But its clear to see they do at least change the agent's behavior, and things like telling it "always use xyz version of java, use gradle to build the project, use this command to run the tests" are really important instead of letting it fumble about trying to find the right thing every…

Yeah I wrote my own language and obviously it’s not in any training data which end tip being an interesting experiment with agents. I’ve found that a few concise skills and an agents.md make a huge difference in guiding the LLMs. Specifically getting them to use the all in one build tool, which the LLMs won’t use without direction.

Re: Do agents.md files help coding agents?

#37

Post by hyperpape »

We need a companion to "IN MICE", which is "IN EVALS".

I don't think this is bad research, but you have to understand how far it generalizes. I'm not saying that evals are useless, we need to do our best to produce good benchmarks. But benchmarks are always going to lag pretty far behind real world applications.

Re: Do agents.md files help coding agents?

#38

Post by hnlmorg »

ch4s3 wrote: post #36
RugnirViking wrote: post #10

yes, they do. I think people overindex on this paper, I remember when it came out we had a lot of discussion in my company about it. But its clear to see they do at least change the agent's behavior, and things like telling it "always use xyz version of java, use gradle to build the project, use this command to run the tests" are really important instead of letting it fumble about trying to find the right thing every…

Yeah I wrote my own language and obviously it’s not in any training data which end tip being an interesting experiment with agents. I’ve found that a few concise skills and an agents.md make a huge difference in guiding the LLMs. Specifically getting them to use the all in one build tool, which the LLMs won’t use without direction.

As a language author myself, I'd be interested to learn more about how you utilised agents.md and skills for this language of yours.

Re: Do agents.md files help coding agents?

#40

Post by sceptic123 »

RugnirViking wrote: post #10

yes, they do. I think people overindex on this paper, I remember when it came out we had a lot of discussion in my company about it. But its clear to see they do at least change the agent's behavior, and things like telling it "always use xyz version of java, use gradle to build the project, use this command to run the tests" are really important instead of letting it fumble about trying to find the right thing every…

That is all the stuff that should be in the README.md of the project in the first place though.
Post reply on HN