Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AGENTS.md outperforms skills in our agent evals

vercel.com

Post reply on HN
Print view View original

41–50 of 212 posts

Re: AGENTS.md outperforms skills in our agent evals

#41

Post by verdverm »

jgbuddy wrote: post #19

Am I missing something here? Obviously directly including context in something like a system prompt will put it in context 100% of the time. You could just as easily take all of an agent's skills, feed it to the agent (in a system prompt, or similar) and it will follow the instructions more reliably. However, at a certain point you have to use skills, because including it in the context every time is wasteful, or not…

You aren't wrong, you really want a bit of both.

1. You absolutely want to force certain context in, no questions or non-determinism asked (index and sparknotes). This can be done conditionally, but still rule based on the files accessed and other "context"

2. You want to keep it clean and only provide useful context as necessary (skills, search, mcp; and really a explore/query/compress mechanism around all of this, ralph wiggum is one example)

Re: AGENTS.md outperforms skills in our agent evals

#42

Post by verdverm »

thorum wrote: post #20

The article presents AGENTS.md as something distinct from Skills, but it is actually a simplified instance of the same concept. Their AGENTS.md approach tells the AI where to find instructions for performing a task. That’s a Skill. I expect the benefit is from better Skill design, specifically, minimizing the number of steps and decisions between the AI’s starting state and the correct information. Fewer transitions…

Yea, I am now separating them based on

1. Those I force into the system prompt using rules based systems and "context"

2. Those I let the agent lookup or discover

I also limit what gets into message parts, moving some of the larger token consumers to the system prompt so they only show once, most notable read/write_file

Re: AGENTS.md outperforms skills in our agent evals

#43

Post by sevg »

cortesoft wrote: post #40
jstummbillig wrote: post #34

Earlier quoted context omitted.

> Obviously directly including context in something like a system prompt will put it in context 100% of the time. How do you suppose skills get announced to the model? It's all in the context in some way. The interesting part here is: Just (relatively naively) compressing stuff in the AGENTS.md seems to work better than however skills are implemented.

Isn't the difference that a skill means you just have to add the script name and explanation to the context instead of the entire script plus the explanation?

You could put the name and explanation in CLAUDE.md/AGENTS.md, plus the path to the rest of the skill that Claude can read if needed.

That seems roughly equivalent to my unenlightened mind!

Re: AGENTS.md outperforms skills in our agent evals

#44

Post by meatcar »

What if instead of needing to run a codemod to cache per-lib docs locally, documentation could be distributed alongside a given lib, as a dev dependency, version locked, and accessible locally as plaintext. All docs can be linked in node_modules/.docs (like binaries are in .bin). It would be a sort of collection of manuals.

What a wonderful world that would be.

Re: AGENTS.md outperforms skills in our agent evals

#45

Post by CuriouslyC »

jryan49 wrote: post #8

Something that I always wonder with each blog post comparing different types of prompt engineering is did they run it once, or multiple times? LLMs are not consistent for the same task. I imagine they realize this of course, but I never get enough details of the testing methodology.

I always make a habit of doing a lot of duplicate runs when I benchmark for this reason. Joke's on me, in the time I spent doing 1 benchmark with real confidence intervals and getting no traction on my post, I could have done 10 shitty benchmarks or 1 shitty benchmark and 9x more blogspam. Perverse incentives rule us all.

Re: AGENTS.md outperforms skills in our agent evals

#46

Post by chr15m »

I'm not sure if this is widely known but you can do a lot better even than AGENTS.md.

Create a folder called .context and symlink anything in there that is relevant to the project. For example READMEs and important docs from dependencies you're using. Then configure your tool to always read .context into context, just like it does for AGENTS.md.

This ensures the LLM has all the information it needs right in context from the get go. Much better performance, cheaper, and less mistakes.

Re: AGENTS.md outperforms skills in our agent evals

#47

Post by mbm »

orlandohohmeier wrote: post #25
jgbuddy wrote: post #19

Am I missing something here? Obviously directly including context in something like a system prompt will put it in context 100% of the time. You could just as easily take all of an agent's skills, feed it to the agent (in a system prompt, or similar) and it will follow the instructions more reliably. However, at a certain point you have to use skills, because including it in the context every time is wasteful, or not…

I’ve been using symlinked agent files for about a year as a hacky workaround before skils became a thing load additional “context” for different tasks, and it might actually address the issue you’re talking about. Honestly, it’s worked so well for me that I haven’t really felt the need to change it.

What sort of files do you generally symlink in?

Re: AGENTS.md outperforms skills in our agent evals

#48

Post by verdverm »

cortesoft wrote: post #40
jstummbillig wrote: post #34

Earlier quoted context omitted.

> Obviously directly including context in something like a system prompt will put it in context 100% of the time. How do you suppose skills get announced to the model? It's all in the context in some way. The interesting part here is: Just (relatively naively) compressing stuff in the AGENTS.md seems to work better than however skills are implemented.

Isn't the difference that a skill means you just have to add the script name and explanation to the context instead of the entire script plus the explanation?

I like to think about it this way, you want to put some high level, table of contents, sparknotes like stuff in the system prompt. This helps warm up the right pathways. In this, you also need to inform that there are more things it may need, depending on "context", through filesystem traversal or search tools, the difference is unimportant, other than most things outside of coding typically don't do filesystem things the same way

Re: AGENTS.md outperforms skills in our agent evals

#50

Post by verdverm »

sheepscreek wrote: post #28

It seems their tests rely on Claude alone. It’s not safe to assume that Codex or Gemini will behave the same way as Claude. I use all three and each has its own idiosyncrasies.

I've done very similar things with my custom agent that uses Gemini and have gotten very similar results. Working on the evals to back that claim up
Post reply on HN