Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AGENTS.md outperforms skills in our agent evals

vercel.com

Post reply on HN
Print view View original

151–160 of 212 posts

Re: AGENTS.md outperforms skills in our agent evals

#151

Post by falcor84 »

gjadi wrote: post #142
BiteCode_dev wrote: post #133

Earlier quoted context omitted.

You got me good with this one. But seriously, this is my main answer to people telling me AI is not reliable: "guess what, most humans are not either, but at least I can tell AI to correct course and it's ego won't get in the way of fixing the problem". In fact, while AI is not nearly as a good as a senior dev for non trivial tasks yet, it is definitely more reliable than most junior devs at following instructions.

It's ego won't get in the way but it's lack of intelligence will. Whereas a junior might be reluctant at first, but if they are smart they will learn and get better. So maybe LLM are better than not-so-smart people, but you usually try to avoid hiring those people in the first place.

That's exactly the thing. Claude Code with Opus 4.5 is already significantly better at essentially everything than a large percentage of devs I had the displeasure of working with, including learning when asked to retain a memory. It's still very far from the best devs, but this is the worse it'll ever be, and it already significantly raised the bar for hiring.

Re: AGENTS.md outperforms skills in our agent evals

#152

Post by baby »

motoboi wrote: post #121

Models are not AGI. They are text generators forced to generate text in a way useful to trigger a harness that will produce effects, like editing files or calling tools. So the model won’t “understand” that you have a skill and use it. The generation of the text that would trigger the skill usage is made via Reinforcement Learning with human generated examples and usage traces. So why don’t the model use skills all t…

I was thinking about that these says and experimenting like so: a system prompt that asks the agent to load any skills that seem relevant early, and a user prompt that asks the agent to do that later when a skill becomes relevant

Re: AGENTS.md outperforms skills in our agent evals

#153

Post by baby »

postalcoder wrote: post #110
w10-1 wrote: post #55

The key finding is that "compression" of doc pointers works. It's barely readable to humans, but directly and efficiently relevant to LLM's (direct reference -> referent, without language verbiage). This suggests some (compressed) index format that is always loaded into context will replace heuristics around agents.md/claude.md/skills.md. So I would bet this year we get some normalization of both the indexes and the…

That's not the only useful takeaway. I found this to be true: > "Explore project first, then invoke skill" [produces better results than] "You MUST invoke the skill". I recently tried to get Antigravity to consistently adhere to my AGENTS.md (Antigravity uses GEMINI.md). The agent consistently ignored instructions in GEMINI.md like: - "You must follow the rules in [..]/AGENTS.md" - "Always refer to your instructions…

ln -s

Re: AGENTS.md outperforms skills in our agent evals

#154

Post by OJFord »

vidarh wrote: post #147
motoboi wrote: post #121

Models are not AGI. They are text generators forced to generate text in a way useful to trigger a harness that will produce effects, like editing files or calling tools. So the model won’t “understand” that you have a skill and use it. The generation of the text that would trigger the skill usage is made via Reinforcement Learning with human generated examples and usage traces. So why don’t the model use skills all t…

> AGENTS.md, on the other hand, is context. Models have been trained to follow context since the dawn of the thing. The skills frontmatter end up in context as well . If AGENTS.md outperform skills in a given agent, it is down to specifically how the skills frontmatter is extracted and injected into the context, because that is the only difference between the two approaches. EDIT: I haven't tried to check this so thi…

No it's more than that - they didn't just put the skills instructions directly in AGENTS.md, they put the whole index for the docs (the skill in this case being a docs lookup) in there, so there's nothing to 'do', the skill output is already in context (or at least pointers to it, the index, if not the actual file contents) not just the front matter.

Hence the submission's conclusion:

> Our working theory [for why this performs better] comes down to three factors.

> No decision point. With AGENTS.md, there's no moment where the agent must decide "should I look this up?" The information is already present.

> Consistent availability. Skills load asynchronously and only when invoked. AGENTS.md content is in the system prompt for every turn.

> No ordering issues. Skills create sequencing decisions (read docs first vs. explore project first). Passive context avoids this entirely.

Re: AGENTS.md outperforms skills in our agent evals

#155

Post by vidarh »

falcor84 wrote: post #151
gjadi wrote: post #142

Earlier quoted context omitted.

It's ego won't get in the way but it's lack of intelligence will. Whereas a junior might be reluctant at first, but if they are smart they will learn and get better. So maybe LLM are better than not-so-smart people, but you usually try to avoid hiring those people in the first place.

That's exactly the thing. Claude Code with Opus 4.5 is already significantly better at essentially everything than a large percentage of devs I had the displeasure of working with, including learning when asked to retain a memory. It's still very far from the best devs, but this is the worse it'll ever be, and it already significantly raised the bar for hiring.

> but this is the worse it'll ever be

And even if the models themselves for some reason were to never get better than what we have now, we've only scratched the surface of harnesses to make them better.

We know a lot about how to make groups of people achieve things individual members never could, and most of the same techiques work for LLMs, but it takes extra work to figure out how to most efficiently work around limitations such as lack of integrated long-term memory.

A lot of that work is in its infancy. E.g. I have a project I'm working on now where I'm up to a couple of dozens of agents, and ever day I'm learning more about how to structure them to squeeze the most out of the models.

One learning that feels relevant to the linked article: Instead of giving an agent the whole task across a large dataset that'd overwhelm context, it often helps to have an agent - that can use Haiku, because it's fine if its dumb - comb the data for , and generate a list of information, and have the bigger model use that as a guide.

So the progress we're seeing is not just raw model improvements, but work like the one in this article: Figuring out how to squeeze the best results out of any given model, and that work would continue to yield improvements for years even if models somehow stopped improving.

Re: AGENTS.md outperforms skills in our agent evals

#156

Post by vidarh »

OJFord wrote: post #154
vidarh wrote: post #147

Earlier quoted context omitted.

> AGENTS.md, on the other hand, is context. Models have been trained to follow context since the dawn of the thing. The skills frontmatter end up in context as well . If AGENTS.md outperform skills in a given agent, it is down to specifically how the skills frontmatter is extracted and injected into the context, because that is the only difference between the two approaches. EDIT: I haven't tried to check this so thi…

No it's more than that - they didn't just put the skills instructions directly in AGENTS.md, they put the whole index for the docs (the skill in this case being a docs lookup) in there, so there's nothing to 'do', the skill output is already in context (or at least pointers to it, the index, if not the actual file contents) not just the front matter. Hence the submission's conclusion: > Our working theory [for why th…

> No it's more than that - they didn't just put the skills instructions directly in AGENTS.md, they put the whole index for the docs (the skill in this case being a docs lookup) in there, so there's nothing to 'do', the skill output is already in context (or at least pointers to it, the index, if not the actual file contents) not just the front matter.

The point remains: That is still just down to how you compose the context/prompt that actually goes to the model.

Nothing stops an agent from including logic to inline the full set of skills if the context is short enough. The point of skills is to provide a mechanism for managing context to reduce the need for summarization/compaction or explicit management, and so allowing you to e.g. have a lot of them available.

(And this kind of makes the article largely moot - it's slightly neat to know it might be better to just inline the skills if you have few enough that they won't seriously fill up your context, but the main value of skills comes when you have enough of them that this isn't the case)

Conversely, nothing prevents the agent from using lossy processing with a smaller, faster model on AGENTS.md either before passing it to the main model e.g. if context is getting out of hand, or if the developer of a given agent think they have a way of making adherence better by transforming them.

These are all tooling decisions, not features of the models.

Re: AGENTS.md outperforms skills in our agent evals

#157

Post by seunosewa »

OJFord wrote: post #154
vidarh wrote: post #147

Earlier quoted context omitted.

> AGENTS.md, on the other hand, is context. Models have been trained to follow context since the dawn of the thing. The skills frontmatter end up in context as well . If AGENTS.md outperform skills in a given agent, it is down to specifically how the skills frontmatter is extracted and injected into the context, because that is the only difference between the two approaches. EDIT: I haven't tried to check this so thi…

No it's more than that - they didn't just put the skills instructions directly in AGENTS.md, they put the whole index for the docs (the skill in this case being a docs lookup) in there, so there's nothing to 'do', the skill output is already in context (or at least pointers to it, the index, if not the actual file contents) not just the front matter. Hence the submission's conclusion: > Our working theory [for why th…

What if they used the same compressed documentation in the skill? That would be just fine too.

Re: AGENTS.md outperforms skills in our agent evals

#158

Post by seanhunter »

cainxinth wrote: post #70
tottenhm wrote: post #5

> In 56% of eval cases, the skill was never invoked. The agent had access to the documentation but didn't use it. The agent passes the Turing test...

Even AI doesn’t RTFM

I can see the future. In a few years, HN will consist entirely of: 1) Bots posting “Show HN” of things they’ve vibecoded

2) Bots replying to those posts,

3) Bots asking whether the bots in #2 even read TFA, and finally

4) Bots posting the HN guideline where it says you shouldn’t ask people whether they have read TFA.

…And amid the smouldering ruins of civilization, the last human, dang, will be there, posting links to all the times this particular thing has been posted to HN before.

Re: AGENTS.md outperforms skills in our agent evals

#159

Post by OJFord »

vidarh wrote: post #156
OJFord wrote: post #154

Earlier quoted context omitted.

No it's more than that - they didn't just put the skills instructions directly in AGENTS.md, they put the whole index for the docs (the skill in this case being a docs lookup) in there, so there's nothing to 'do', the skill output is already in context (or at least pointers to it, the index, if not the actual file contents) not just the front matter. Hence the submission's conclusion: > Our working theory [for why th…

> No it's more than that - they didn't just put the skills instructions directly in AGENTS.md, they put the whole index for the docs (the skill in this case being a docs lookup) in there, so there's nothing to 'do', the skill output is already in context (or at least pointers to it, the index, if not the actual file contents) not just the front matter. The point remains: That is still just down to how you compose the…

However you compose the context for the skill, the model has to generate output like 'use skill docslookup(blah)' vs. just 'according to the docs in context' (or even 'read file blah.txt mentioned in context') which training can affect.

Re: AGENTS.md outperforms skills in our agent evals

#160

Post by OJFord »

seunosewa wrote: post #157
OJFord wrote: post #154

Earlier quoted context omitted.

No it's more than that - they didn't just put the skills instructions directly in AGENTS.md, they put the whole index for the docs (the skill in this case being a docs lookup) in there, so there's nothing to 'do', the skill output is already in context (or at least pointers to it, the index, if not the actual file contents) not just the front matter. Hence the submission's conclusion: > Our working theory [for why th…

What if they used the same compressed documentation in the skill? That would be just fine too.

Sure but it would be a trivial comparison then, this is really about context vs tool-calling.
Post reply on HN