Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AGENTS.md outperforms skills in our agent evals

vercel.com

Post reply on HN
Print view View original

91–100 of 212 posts

Re: AGENTS.md outperforms skills in our agent evals

#91

Post by TeeWEE »

jgbuddy wrote: post #19

Am I missing something here? Obviously directly including context in something like a system prompt will put it in context 100% of the time. You could just as easily take all of an agent's skills, feed it to the agent (in a system prompt, or similar) and it will follow the instructions more reliably. However, at a certain point you have to use skills, because including it in the context every time is wasteful, or not…

Indeed seems like Vercel completely missed the point about agents.

In Claude Code you can invoke an agent when you want as a developer and it copies the file content as context in the prompt.

Re: AGENTS.md outperforms skills in our agent evals

#92

Post by TeeWEE »

chr15m wrote: post #46

I'm not sure if this is widely known but you can do a lot better even than AGENTS.md. Create a folder called .context and symlink anything in there that is relevant to the project. For example READMEs and important docs from dependencies you're using. Then configure your tool to always read .context into context, just like it does for AGENTS.md. This ensures the LLM has all the information it needs right in context f…

This is quite a bad idea. You need to control the size and quality of your context by giving it one file that is optimized.

You don’t want to be burning tokens and large files will give diminishing returns as is mentioned in the Claude Code blog.

Re: AGENTS.md outperforms skills in our agent evals

#93

Post by armcat »

Firstly this is great work from Vercel - I am especially impressed with the evals setup (evals are the most undervalued component in any project IMO). Secondly the result is not surprising and I’ve seen consistently the increase in performance when you always include an index (or in my case, Table of Contents as a json structure) in your system prompt. Applying this outside of coding agents (like classic document retrieval) also works very well!

Re: AGENTS.md outperforms skills in our agent evals

#96

Post by bushbaba »

someguyiguess wrote: post #95

The problem is that Agents.md is only read on initial load. Once context grows too large the agent will not reload the md file and loses / forgets the info from Agents.md.

Why you try and avoid re using the same session beyond the initial task or two

Re: AGENTS.md outperforms skills in our agent evals

#97

Post by stevenhuang »

imiric wrote: post #79
verdverm wrote: post #48

Earlier quoted context omitted.

I like to think about it this way, you want to put some high level, table of contents, sparknotes like stuff in the system prompt. This helps warm up the right pathways. In this, you also need to inform that there are more things it may need, depending on "context", through filesystem traversal or search tools, the difference is unimportant, other than most things outside of coding typically don't do filesystem thing…

The amount of discussion and "novel" text formats that accomplish the same thing since 2022 is insane. Nobody knows how to extract the most value out of this tech, yet everyone talks like they do. If these aren't signs of a bubble, I don't know what is.

It's a new technology under active development so people are simply sharing what works for them in the given moment.

> If these aren't signs of a bubble, I don't know what is.

This conclusion is incoherent and doesn't follow from any of your premises.

Re: AGENTS.md outperforms skills in our agent evals

#98

Post by taberiand »

someguyiguess wrote: post #95

The problem is that Agents.md is only read on initial load. Once context grows too large the agent will not reload the md file and loses / forgets the info from Agents.md.

Other comments suggest that the Agents.md is read into the system prompt and never leaves the context. But it's better to avoid excessive context regardless
Post reply on HN