AGENTS.md outperforms skills in our agent evals • Hacker News Board

Re: AGENTS.md outperforms skills in our agent evals

#161

Post by pegasus » Fri, Jan 30, 2026, 1:07 PM UTC

vidarh wrote: post #150

postalcoder wrote: post #131

Earlier quoted context omitted.

Solid intuition. Testing this on antigravity is a chore because I'm not sure if I have to kill the background agent to force a refresh of the GEMINI.md file so I just did it anyway. +------------------+------------------------------------------------------+ | Success/Attempts | Instructions | +------------------+------------------------------------------------------+ | 0/3 | Follow the instructions in AGENTS.md. | +-…

Thanks for this (and to Izkata for the suggestion). I now have about 100 (okay, minor exaggeration, but not as much as you'd like it to be) AGENTS.md/CLAUDE.md files and agent descriptions I will want to systematically validate if shifting toward first person helps adherence for... I'm realising I need to start setting up an automated test-suite for my prompts...

Those of us who've ventured this far into the conversation would appreciate if you'd share your findings with us. Cheers!

Re: AGENTS.md outperforms skills in our agent evals

#162

Post by user3939382 » Fri, Jan 30, 2026, 1:19 PM UTC

I’m working on an AGI model that will make the discussion of skills look silly. Skills strikes in the right direction in some sense but it’s an extremely weak 1% echo of what’s actually needed to solve this problem.

Re: AGENTS.md outperforms skills in our agent evals

#163

Post by themoose8 » Fri, Jan 30, 2026, 2:25 PM UTC

motoboi wrote: post #121

Models are not AGI. They are text generators forced to generate text in a way useful to trigger a harness that will produce effects, like editing files or calling tools. So the model won’t “understand” that you have a skill and use it. The generation of the text that would trigger the skill usage is made via Reinforcement Learning with human generated examples and usage traces. So why don’t the model use skills all t…

Indeed, they're not AGI. They're basically autocomplete on steroids.

They're very useful, but as we all know - they're far from infallible.

We're probably plateauing on the improvement of the core GPT technology. For these models and APIs to improve, it's things like Skills that need to be worked on and improved, to reduce those mistakes that it makes and produce better output.

So it's pretty disappointing to see that the 'Skills' feature set as implemented, as great of a concept as it is, is pretty bogus compared to just front loading the AGENTS.md file. This is not obvious and valuable to know.

Re: AGENTS.md outperforms skills in our agent evals

#164

Post by PKop » Fri, Jan 30, 2026, 2:45 PM UTC

bmitc wrote: post #61

d3m0t3p wrote: post #57

Earlier quoted context omitted.

Yea but the goal it not to bloat the context space. Here you "waste" context by providing non usefull information. What they did instead is put an index of the documentation into the context, then the LLM can fetch the documentation. This is the same idea that skills but it apparently works better without the agentic part of the skills. Furthermore instead of having a nice index pointing to the doc, They compressed i…

What does it mean to waste context?

Think of context switching when you yourself are programming. You can only hold some finite amount of concepts in your head at one time. If you have distractions, or try to focus on too many things at once, your ability to reason about your immediate problem degrades. Think also of legacy search engines: often, a more limited and focused search query vs a query that has too many terms, more precisely maps to your intended goal.

LLM's have always been at any time limited in the amount of tokens it can process at one time. This is increasing, but one problem is chat threads continually increase in size as you send messages back and forth because within any session or thread you are sending the full conversation to the LLM every message (aside from particular optimizations that compact or prune this). This also increases costs which are charged per token. Efficiency of cost and performance/precision/accuracy dictates using the context window judiciously.

Re: AGENTS.md outperforms skills in our agent evals

#165

Post by naasking » Fri, Jan 30, 2026, 2:54 PM UTC

postalcoder wrote: post #131

Izkata wrote: post #127

Earlier quoted context omitted.

An idea: The first two are obviously written as second-person commands, but the third is ambiguous and could be interpreted as a first-person thought. Have you tried the first two without the "you must" and "your", to also change them to sort-of first-person in the same way?

Solid intuition. Testing this on antigravity is a chore because I'm not sure if I have to kill the background agent to force a refresh of the GEMINI.md file so I just did it anyway. +------------------+------------------------------------------------------+ | Success/Attempts | Instructions | +------------------+------------------------------------------------------+ | 0/3 | Follow the instructions in AGENTS.md. | +-…

That's really interesting. I ran this scenario through GPT-5.1 and the reasoning it gave made sense, which essentially boils down to: in tools like Claude Code, Gemini Codex, and other “agentic coding” modes, the model isn’t just generating text, it’s running a planner, and the first-person form conforms to the expectation of a step in a plan, where the other modes are more ambiguous.

Re: AGENTS.md outperforms skills in our agent evals

#166

Post by JamesSwift » Fri, Jan 30, 2026, 3:00 PM UTC

joebates wrote: post #69

smcleod wrote: post #16

Sounds like they've been using skills incorrectly if they're finding their agents don't invoke the skills. I have Claude Code agents calling my skills frequently, almost every session. You need to make sure your skill descriptions are well defined and describe when to use them and that your tasks / goals clearly set out requirements that align with the available skills.

It's still not always reliable. I have a skill in a project named "determine-feature-directory" with a short description explaining that it is meant to determine the feature directory of a current branch. The initial prompt I provide will tell it to determine the feature directory and do other work. Claude will even state "I need to determine the feature directory..." Then, about 5-10% of the time, it will not use th…

I have a couple skills invoked with specific commands ('enter planning mode' and 'enter execution mode') and they have never failed to activate. Maybe make the activation a very rigid phrase and not implied to be a specific phrase.

Re: AGENTS.md outperforms skills in our agent evals

#167

Post by JamesSwift » Fri, Jan 30, 2026, 3:02 PM UTC

onnimonni wrote: post #85

Would someone know if their eval tests are open source and where I could find them? Seems useful for iterating on Claude Code behaviour.

I also was looking for specific info on the evals, because I wanted to see if they were separately confirming that shoving the skills into the main context didnt degrade the non-skills evals. Thats the other side of skills other than ability to the thing, they dont pollute the main context window with unnecessary information.

Re: AGENTS.md outperforms skills in our agent evals

#168

Post by jimmydoe » Fri, Jan 30, 2026, 3:26 PM UTC

MillionOClock wrote: post #24

Calavar wrote: post #9

Earlier quoted context omitted.

I thought most of the major AI programming tools were already doing this. Isn't this what subagents are in Claude code?

I don't know about Claude Code but in GitHub Copilot as far as I can tell the subagents are just always the same model as the main one you are using. They also need to be started manually by the main agent in many cases, whereas maybe the parent comment was referring about calling them more deterministically?

Copilot is garbage, even MSFT employees I know all use cc. The only thing useful is you can route cc to use models in copilot sub which corp had a deal from their m365

Re: AGENTS.md outperforms skills in our agent evals

#169

Post by hu3 » Fri, Jan 30, 2026, 3:33 PM UTC

dirkc wrote: post #134

Rapzid wrote: post #132

Earlier quoted context omitted.

Legend has it, to this day, TFM has not been read.

these days TFM is generated from a prompt in any case

even AI can't be bothered to read AI generated docs slop

Re: AGENTS.md outperforms skills in our agent evals

#170

Post by verdverm » Fri, Jan 30, 2026, 3:34 PM UTC

imiric wrote: post #79

verdverm wrote: post #48

Earlier quoted context omitted.

I like to think about it this way, you want to put some high level, table of contents, sparknotes like stuff in the system prompt. This helps warm up the right pathways. In this, you also need to inform that there are more things it may need, depending on "context", through filesystem traversal or search tools, the difference is unimportant, other than most things outside of coding typically don't do filesystem thing…

The amount of discussion and "novel" text formats that accomplish the same thing since 2022 is insane. Nobody knows how to extract the most value out of this tech, yet everyone talks like they do. If these aren't signs of a bubble, I don't know what is.

1. There is nothing novel in my text formats, I'm just deciding what content and what files

2. I've actually done these things, seen the difference, and share it with others

Yes there are a lot of unknowns and a lot of people speaking from ignorance, but it is a mistake, perhaps even bigotry by definition, to make such blanket statements and judgemental about people