Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AGENTS.md outperforms skills in our agent evals

vercel.com

Post reply on HN
Print view View original

71–80 of 212 posts

Re: AGENTS.md outperforms skills in our agent evals

#71

Post by deaux »

jgbuddy wrote: post #19

Am I missing something here? Obviously directly including context in something like a system prompt will put it in context 100% of the time. You could just as easily take all of an agent's skills, feed it to the agent (in a system prompt, or similar) and it will follow the instructions more reliably. However, at a certain point you have to use skills, because including it in the context every time is wasteful, or not…

You're right, the results are completely as expected.

The article also doesn't mention that they don't know how the compressed index output quality. That's always a concern with this kind of compression. Skills are just another, different kind of compression. One with a much higher compression rate and presumably less likely to negatively influence quality. The cost being that it doesn't always get invoked.

Re: AGENTS.md outperforms skills in our agent evals

#73

Post by ethmarks »

ai-christianson wrote: post #66
w10-1 wrote: post #55

The key finding is that "compression" of doc pointers works. It's barely readable to humans, but directly and efficiently relevant to LLM's (direct reference -> referent, without language verbiage). This suggests some (compressed) index format that is always loaded into context will replace heuristics around agents.md/claude.md/skills.md. So I would bet this year we get some normalization of both the indexes and the…

They say compressed... but isn't this just "minified"?

Minification is still a form of compression, it just leaves the file more readable than more powerful compression methods (such as ZIP archives).

Re: AGENTS.md outperforms skills in our agent evals

#78

Post by denolfe »

PreSession Hook from obra/superpowers injects this along with more logic for getting rid of rationalizing out of using skills:

> If you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill. IF A SKILL APPLIES TO YOUR TASK, YOU DO NOT HAVE A CHOICE. YOU MUST USE IT.

While this may result in overzealous activation of skills, I've found that if I have a skill related, I _want_ to use it. It has worked well for me.

Re: AGENTS.md outperforms skills in our agent evals

#79

Post by imiric »

verdverm wrote: post #48
cortesoft wrote: post #40

Earlier quoted context omitted.

Isn't the difference that a skill means you just have to add the script name and explanation to the context instead of the entire script plus the explanation?

I like to think about it this way, you want to put some high level, table of contents, sparknotes like stuff in the system prompt. This helps warm up the right pathways. In this, you also need to inform that there are more things it may need, depending on "context", through filesystem traversal or search tools, the difference is unimportant, other than most things outside of coding typically don't do filesystem thing…

The amount of discussion and "novel" text formats that accomplish the same thing since 2022 is insane. Nobody knows how to extract the most value out of this tech, yet everyone talks like they do. If these aren't signs of a bubble, I don't know what is.
Post reply on HN