Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AGENTS.md outperforms skills in our agent evals

vercel.com

Post reply on HN
Print view View original

121–130 of 212 posts

Re: AGENTS.md outperforms skills in our agent evals

#121

Post by motoboi »

Models are not AGI. They are text generators forced to generate text in a way useful to trigger a harness that will produce effects, like editing files or calling tools.

So the model won’t “understand” that you have a skill and use it. The generation of the text that would trigger the skill usage is made via Reinforcement Learning with human generated examples and usage traces.

So why don’t the model use skills all the time? Because it’s a new thing, there is not enough training samples displaying that behavior.

They also cannot enforce that via RL because skills use human language, which is ambiguous and not formal. Force it to use skills always via RL policy and you’ll make the model dumber.

So, right now, we are generating usage traces that will be used to train the future models to get a better grasp of when to use skills not. Just give it time.

AGENTS.md, on the other hand, is context. Models have been trained to follow context since the dawn of the thing.

Re: AGENTS.md outperforms skills in our agent evals

#122

Post by deaux »

devonkelley wrote: post #116
deaux wrote: post #103

Earlier quoted context omitted.

This comment instantly set off my LLM alarm bells. Went into the profile, and guess what: next comment (not a one-liner) [0] on a completely different topic was posted 35 seconds later. And includes the classic "aren't just A. They're B.". Why are you doing this? Karma? 8 years old account and first post 3 days ago is a Show HN shilling your "AI agent" SaaS with a boatload of fake comments? [1] Pinging tomhow [0] htt…

Dude I am not AI. Real human. Just started on HN.

Just happen to post 2 comments within 30s on completely different posts, having all of the hallmarks of LLM output? With your other post being full of green accounts? With no account activity for 8 years? You're clearly posting comments straight from an LLM.

It's not realistic to read the other post to a significant degree, think about it, and then type all of this:

> The prompt injection concerns are valid, but I think there's a more fundamental issue: agents are non-deterministic systems that fail in ways that are hard to predict or debug. Security is one failure mode. But "agent did something subtly wrong that didn't trigger any errors" is another. And unlike a hacked system where you notice something's off, a flaky agent just... occasionally does the wrong thing. Sometimes it works. Sometimes it doesn't. Figuring out which case you're in requires building the same observability infrastructure you'd use for any unreliable distributed system.

> The people running these connected to their email or filesystem aren't just accepting prompt injection risk. They're accepting that their system will randomly succeed or fail at tasks depending on model performance that day, and they may not notice the failures until later.

Within 35 seconds of posting this one. And it just happens to have all LLM hallmarks there are. We both know it, you're on HN, people here aren't fools.

Re: AGENTS.md outperforms skills in our agent evals

#123

Post by devonkelley »

deaux wrote: post #122
devonkelley wrote: post #116

Earlier quoted context omitted.

Dude I am not AI. Real human. Just started on HN.

Just happen to post 2 comments within 30s on completely different posts, having all of the hallmarks of LLM output? With your other post being full of green accounts? With no account activity for 8 years? You're clearly posting comments straight from an LLM. It's not realistic to read the other post to a significant degree, think about it, and then type all of this: > The prompt injection concerns are valid, but I th…

I made an account years ago, never posted, and decided I want to be more active in the community.

Green accounts probably bc I sent my post to some friends and users directly when I made it. Is that illegal on HN? I legit don't know how things work here. I was excited over my launch post.

Anyways, not a fucking bot, my company is real, the commenters on my post are real and if it's a crime for me to rapid fire post and/or have friends comment on my Show HN, good to know.

Re: AGENTS.md outperforms skills in our agent evals

#125

Post by bzGoRust »

motoboi wrote: post #121

Models are not AGI. They are text generators forced to generate text in a way useful to trigger a harness that will produce effects, like editing files or calling tools. So the model won’t “understand” that you have a skill and use it. The generation of the text that would trigger the skill usage is made via Reinforcement Learning with human generated examples and usage traces. So why don’t the model use skills all t…

I completed agree with your point

Re: AGENTS.md outperforms skills in our agent evals

#127

Post by Izkata »

postalcoder wrote: post #110
w10-1 wrote: post #55

The key finding is that "compression" of doc pointers works. It's barely readable to humans, but directly and efficiently relevant to LLM's (direct reference -> referent, without language verbiage). This suggests some (compressed) index format that is always loaded into context will replace heuristics around agents.md/claude.md/skills.md. So I would bet this year we get some normalization of both the indexes and the…

That's not the only useful takeaway. I found this to be true: > "Explore project first, then invoke skill" [produces better results than] "You MUST invoke the skill". I recently tried to get Antigravity to consistently adhere to my AGENTS.md (Antigravity uses GEMINI.md). The agent consistently ignored instructions in GEMINI.md like: - "You must follow the rules in [..]/AGENTS.md" - "Always refer to your instructions…

An idea: The first two are obviously written as second-person commands, but the third is ambiguous and could be interpreted as a first-person thought. Have you tried the first two without the "you must" and "your", to also change them to sort-of first-person in the same way?

Re: AGENTS.md outperforms skills in our agent evals

#128

Post by chr15m »

TeeWEE wrote: post #92
chr15m wrote: post #46

I'm not sure if this is widely known but you can do a lot better even than AGENTS.md. Create a folder called .context and symlink anything in there that is relevant to the project. For example READMEs and important docs from dependencies you're using. Then configure your tool to always read .context into context, just like it does for AGENTS.md. This ensures the LLM has all the information it needs right in context f…

This is quite a bad idea. You need to control the size and quality of your context by giving it one file that is optimized. You don’t want to be burning tokens and large files will give diminishing returns as is mentioned in the Claude Code blog.

It is not an "idea" but something I've been doing for months and it works very well. YMMV. Yes, you should avoid large files and control the size and quality of your context.

Re: AGENTS.md outperforms skills in our agent evals

#129

Post by singingbard »

jgbuddy wrote: post #19

Am I missing something here? Obviously directly including context in something like a system prompt will put it in context 100% of the time. You could just as easily take all of an agent's skills, feed it to the agent (in a system prompt, or similar) and it will follow the instructions more reliably. However, at a certain point you have to use skills, because including it in the context every time is wasteful, or not…

So you’re not missing anything if you use Claude by yourself. You just update your local system prompt.

Instead it’s a problem when you’re part of a team and you’re using skills for standards like code style or architectural patterns. You can’t ask everyone to constantly update their system prompt.

Claude skill adherence is very low.

Re: AGENTS.md outperforms skills in our agent evals

#130

Post by chr15m »

gbnwl wrote: post #114
chr15m wrote: post #86

Earlier quoted context omitted.

As they point out in the article, that approach is fragile. Cheaper because it has the right context from the start instead of faffing about trying to find it, which uses tokens and ironically bloats context. It doesn't have to be every bit of documentation, but putting the most salient bits in context makes LLMs perform much more efficiently and accurately in my experience. You can also use the trick of asking an LL…

> Extracting the most useful parts of documentation into a file Yes, and this file becomes: also documentation. I didn’t mean throw entire unabridged docs at it, I should’ve been more clear. All of my docs for agents are written by agents themselves. Either way once the project becomes sufficiently complex it’s just not going to be feasible to add a useful level of detail of every part of it into context by default,…

> sufficiently complex

Sounds like we are working on different types of projects. I avoid complexity at almost all cost and ruthlessly minimise LoC and infrastructure. I realise that's a privilege, and many programmers can't.

Post reply on HN