Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AGENTS.md outperforms skills in our agent evals

vercel.com

Post reply on HN
Print view View original

201–210 of 212 posts

Re: AGENTS.md outperforms skills in our agent evals

#201

Post by js8 »

blueprint wrote: post #191
js8 wrote: post #188

Earlier quoted context omitted.

> Models are not AGI. How do you know? What if AGI can be implemented as a reasonably small set of logic rules, which implement what we call "epistemology" and "informal reasoning"? And this set of rules is just being run in a loop, producing better and better models of reality. It might even include RL, for what we know. And what if LLMs already know all these rules? So they are AGI-complete without us knowing. To b…

It's very simple. The model itself doesn't know and can't verify it. It knows that it doesn't know. Do you deny that? Or do you think that a general intelligence would be in the habit of lying to people and concealing why? At the end of the day, that would be not only unintelligent, but hostile. So it's very simple. And there is such a thing as "the truth", and it can be verified by anyone repeatably in the requisite…

All I asked for was the OP to substantiate their claim that LLMs are not AGI. I am agnostic on that - either way seems plausible.

I don't think there even is an agreed criterion of what AGI is. Current models can easily pass the Turing test (except some gotchas, but these don't really test intelligence).

Re: AGENTS.md outperforms skills in our agent evals

#202

Post by inadequatespace »

postalcoder wrote: post #110
w10-1 wrote: post #55

The key finding is that "compression" of doc pointers works. It's barely readable to humans, but directly and efficiently relevant to LLM's (direct reference -> referent, without language verbiage). This suggests some (compressed) index format that is always loaded into context will replace heuristics around agents.md/claude.md/skills.md. So I would bet this year we get some normalization of both the indexes and the…

That's not the only useful takeaway. I found this to be true: > "Explore project first, then invoke skill" [produces better results than] "You MUST invoke the skill". I recently tried to get Antigravity to consistently adhere to my AGENTS.md (Antigravity uses GEMINI.md). The agent consistently ignored instructions in GEMINI.md like: - "You must follow the rules in [..]/AGENTS.md" - "Always refer to your instructions…

Interesting. It's almost like models don't like being ordered around rudely with this "must” language.

Perhaps what they've learned from training data is “must” often occurs in cases with bullshit red tape or other regulations. "You must read the terms and conditions before using this stuff," or something like that, which are actually best ignored.

Re: AGENTS.md outperforms skills in our agent evals

#203

Post by MillionOClock »

jimmydoe wrote: post #168
MillionOClock wrote: post #24

Earlier quoted context omitted.

I don't know about Claude Code but in GitHub Copilot as far as I can tell the subagents are just always the same model as the main one you are using. They also need to be started manually by the main agent in many cases, whereas maybe the parent comment was referring about calling them more deterministically?

Copilot is garbage, even MSFT employees I know all use cc. The only thing useful is you can route cc to use models in copilot sub which corp had a deal from their m365

On of the advantages of GitHub Copilot for me is that in terms of billing I find it very generous, depending on how you use it.

Re: AGENTS.md outperforms skills in our agent evals

#204

Post by blueprint »

js8 wrote: post #201
blueprint wrote: post #191

Earlier quoted context omitted.

It's very simple. The model itself doesn't know and can't verify it. It knows that it doesn't know. Do you deny that? Or do you think that a general intelligence would be in the habit of lying to people and concealing why? At the end of the day, that would be not only unintelligent, but hostile. So it's very simple. And there is such a thing as "the truth", and it can be verified by anyone repeatably in the requisite…

All I asked for was the OP to substantiate their claim that LLMs are not AGI. I am agnostic on that - either way seems plausible. I don't think there even is an agreed criterion of what AGI is. Current models can easily pass the Turing test (except some gotchas, but these don't really test intelligence).

What people hope 'AGI' is would at least be able to make confirmations of fact and know what verification means. LLMs don't have 'knowledge' and do not actually 'reason'. Heuristic vs simulation. One can be made to approach the other, but only on a specific and narrow path. Someone who knows something can verify that they know it. An "intelligence" implies it is doing operations based on rules, but LLMs cannot conform themselves to rules that require them to reason everything through. What people have hoped AGI would be could be trained to reliably adopt the practice of reasoning. Necessary but maybe not sufficient, and I'm just gonna blame that on the term "intelligence" actually indicating a still relatively low level of what I will "consciousness".

Re: AGENTS.md outperforms skills in our agent evals

#205

Post by js8 »

blueprint wrote: post #204
js8 wrote: post #201

Earlier quoted context omitted.

All I asked for was the OP to substantiate their claim that LLMs are not AGI. I am agnostic on that - either way seems plausible. I don't think there even is an agreed criterion of what AGI is. Current models can easily pass the Turing test (except some gotchas, but these don't really test intelligence).

What people hope 'AGI' is would at least be able to make confirmations of fact and know what verification means. LLMs don't have 'knowledge' and do not actually 'reason'. Heuristic vs simulation. One can be made to approach the other, but only on a specific and narrow path. Someone who knows something can verify that they know it. An "intelligence" implies it is doing operations based on rules, but LLMs cannot confor…

I don't really follow what you're saying, so I'll keep it short. I have used Claude Opus 4.5 for coding and it certainly has knowledge and can reason.

You're wrong on reliability. Humans are also quite unreliable, and formal reasoning systems in silico can actually fail too (due to e.g. cosmic rays), the probability is just astronomically low.

And in engineering, we know quite well how to take a system that is less than 50% unreliable and turn it into something with any degree of reliability - we just run it over and over and verify it gives identical results.

And Claude Code (as an LLM harness) can do this. It can write tests. It can check if program is running correctly (giving expected result). It can be made to any degree of reliability you desire. We've crossed that 50% threshold.

The same happens when models are learning. They start with heuristics, but eventually they'll learn and generalize enough to learn whatever formal rules of logic and reasoning, and to apply them with high degree of reliability. Again, we've probably crossed that threshold, which is confirmed by experience of many users that models are getting more and more reliable with each iteration.

Does it make me uneasy that I don't know what the underlying learned formal reasoning system is? Yes. But that doesn't mean it's not AGI.

Re: AGENTS.md outperforms skills in our agent evals

#206

Post by aktau »

verdverm wrote: post #38

This largely mirrors my experience building my custom agent 1. Start from the Claude Code extracted instructions, they have many things like this in there. Their knowledge share in docs and blog on this aspect are bar none 2. Use AGENTS.md as a table of contents and sparknotes, put them everywhere, load them automatically 3. Have topical markdown files / skills 4. Make great tools, this is still opaque in my mind to…

> I changed read/write_file to put contents in the state and presented in the system prompt, same for the agents.md, now working on evals to show how much better this is, because anecdotally, it kicks ass.

Can you detail this a bit more? Do you put the actual contents of the file in the system prompt? Forever?

Re: AGENTS.md outperforms skills in our agent evals

#207

Post by blueprint »

js8 wrote: post #205
blueprint wrote: post #204

Earlier quoted context omitted.

What people hope 'AGI' is would at least be able to make confirmations of fact and know what verification means. LLMs don't have 'knowledge' and do not actually 'reason'. Heuristic vs simulation. One can be made to approach the other, but only on a specific and narrow path. Someone who knows something can verify that they know it. An "intelligence" implies it is doing operations based on rules, but LLMs cannot confor…

I don't really follow what you're saying, so I'll keep it short. I have used Claude Opus 4.5 for coding and it certainly has knowledge and can reason. You're wrong on reliability. Humans are also quite unreliable, and formal reasoning systems in silico can actually fail too (due to e.g. cosmic rays), the probability is just astronomically low. And in engineering, we know quite well how to take a system that is less t…

> It can be made to any degree of reliability you desire.

Absolutely false statement.

Re: AGENTS.md outperforms skills in our agent evals

#209

Post by farhanhubble »

So the root cause was the model's indisposition to calling the skills. That seems contrary to what we see with function calling. Models call functions quite reliably most of the time. This is more likely because of the instructions not being clear about what skills are, as this snippet, albeit in isolation, seems to suggest:

> Before writing code, first explore the project structure, then invoke the nextjs-doc skill for documentation.

Re: AGENTS.md outperforms skills in our agent evals

#210

Post by alex_metacraft »

postalcoder wrote: post #131
Izkata wrote: post #127

Earlier quoted context omitted.

An idea: The first two are obviously written as second-person commands, but the third is ambiguous and could be interpreted as a first-person thought. Have you tried the first two without the "you must" and "your", to also change them to sort-of first-person in the same way?

Solid intuition. Testing this on antigravity is a chore because I'm not sure if I have to kill the background agent to force a refresh of the GEMINI.md file so I just did it anyway. +------------------+------------------------------------------------------+ | Success/Attempts | Instructions | +------------------+------------------------------------------------------+ | 0/3 | Follow the instructions in AGENTS.md. | +-…

This is a really interesting finding. It makes sense when you think about what the training data looks like — first person statements in a system prompt pattern-match to "internal monologue" or "chain of thought" examples, which the model has been heavily trained to follow through on. Second person commands pattern-match to user instructions, which the model has also been trained to sometimes push back on or reinterpret.

There's probably a related effect with imperative vs. declarative framing in skills too. "When the user asks about X, do Y" seems to work worse than "This project uses Y for X" in my experience. The declarative version reads like a fact about the world rather than a command to obey, and models seem to treat facts as more reliable context.

Would be curious if someone has tested this systematically across different models. The optimal framing might vary quite a bit between Claude, Gemini, and GPT.

Post reply on HN