Blackbox oracles make bad workflows, and tend to produce a whole lot of cargo culting. It's this kind of opacity (why does the markdown outperform agents? there's no real way to find out, even with a fully open or house model because the nature of the beast is that the execution path in a model can't be predicted) that makes me shy away from saying LLMs are "just another tool". If I can't see inside it -- and if even the vendor can't really see inside of it -- there's something fundamentally different.
AGENTS.md outperforms skills in our agent evals
101–110 of 212 posts
Re: AGENTS.md outperforms skills in our agent evals
#102Am I missing something here? Obviously directly including context in something like a system prompt will put it in context 100% of the time. You could just as easily take all of an agent's skills, feed it to the agent (in a system prompt, or similar) and it will follow the instructions more reliably. However, at a certain point you have to use skills, because including it in the context every time is wasteful, or not…
> Obviously directly including context in something like a system prompt will put it in context 100% of the time. How do you suppose skills get announced to the model? It's all in the context in some way. The interesting part here is: Just (relatively naively) compressing stuff in the AGENTS.md seems to work better than however skills are implemented.
Skills have frontmatter which includes a name and description. The description is what determines if the llm finds the skill useful for the task at hand.
If your agent isn’t being used, it’s not as simple as “agents aren’t getting called”. You have to figure out how to get the agent invoked.
Re: AGENTS.md outperforms skills in our agent evals
#103[flagged]
This comment instantly set off my LLM alarm bells. Went into the profile, and guess what: next comment (not a one-liner) [0] on a completely different topic was posted 35 seconds later. And includes the classic "aren't just A. They're B.".
Why are you doing this? Karma? 8 years old account and first post 3 days ago is a Show HN shilling your "AI agent" SaaS with a boatload of fake comments? [1]
Pinging tomhow
Re: AGENTS.md outperforms skills in our agent evals
#104I did a similar set of evals myself utilising the baseline capabilities that Phoenix (elixir) ships with and then skillified them.
Regularly the skills were not being loaded and thus not utilised. The outputs themselves were fine. This suggested that at some stage through the improvements of the models that baseline AGENTS.md had become redundant.
Re: AGENTS.md outperforms skills in our agent evals
#105The key finding is that "compression" of doc pointers works. It's barely readable to humans, but directly and efficiently relevant to LLM's (direct reference -> referent, without language verbiage). This suggests some (compressed) index format that is always loaded into context will replace heuristics around agents.md/claude.md/skills.md. So I would bet this year we get some normalization of both the indexes and the…
Would’ve been perfectly readable and no larger if they had used newline instead of pipe.
Re: AGENTS.md outperforms skills in our agent evals
#106[flagged]
This comment instantly set off my LLM alarm bells. Went into the profile, and guess what: next comment (not a one-liner) [0] on a completely different topic was posted 35 seconds later. And includes the classic "aren't just A. They're B.". Why are you doing this? Karma? 8 years old account and first post 3 days ago is a Show HN shilling your "AI agent" SaaS with a boatload of fake comments? [1] Pinging tomhow [0] htt…
Wow.
Re: AGENTS.md outperforms skills in our agent evals
#107Re: AGENTS.md outperforms skills in our agent evals
#108Earlier quoted context omitted.
> Obviously directly including context in something like a system prompt will put it in context 100% of the time. How do you suppose skills get announced to the model? It's all in the context in some way. The interesting part here is: Just (relatively naively) compressing stuff in the AGENTS.md seems to work better than however skills are implemented.
Skills have frontmatter which includes a name and description. The description is what determines if the llm finds the skill useful for the task at hand. If your agent isn’t being used, it’s not as simple as “agents aren’t getting called”. You have to figure out how to get the agent invoked.
Sure, but then you're playing a very annoying and boring game of model-whispering to specific versions of models that are ever changing as well as trying to hopefully get it to respond correctly with who knows what user input surrounds it.
I really only think the game is worth playing when it's against a fixed version of a specific model. The amount of variance we observe between different releases of the same model is enough to require us to update our prompts and re-test. I don't envy anyone who has to try and find some median text that performs okay on every model.
Re: AGENTS.md outperforms skills in our agent evals
#109This doesn't surprise me.
I have a SKILL.md for marimo notebooks with instructions in the frontmatter to always read it before working with marimo files. But half the time Claude Code still doesn't invoke it even with me mentioning marimo in the first conversation turn.
I've resorted to typing "read marimo skill" manually and that works fine. Technically you can use skills with slash commands but that automatically sends off the message too which just wastes time.
But the actual concept of instructions to load in certain scenarios is very good and has been worth the time to write up the skill.
Re: AGENTS.md outperforms skills in our agent evals
#110The key finding is that "compression" of doc pointers works. It's barely readable to humans, but directly and efficiently relevant to LLM's (direct reference -> referent, without language verbiage). This suggests some (compressed) index format that is always loaded into context will replace heuristics around agents.md/claude.md/skills.md. So I would bet this year we get some normalization of both the indexes and the…
That's not the only useful takeaway. I found this to be true:
> "Explore project first, then invoke skill" [produces better results than] "You MUST invoke the skill".
- "You must follow the rules in [..]/AGENTS.md"
- "Always refer to your instructions in [..]/AGENTS.md"
Yet, this works every time: "Check for the presence of AGENTS.md files in the project workspace."
This behavior is mysterious. It's like how, in earlier days, "let's think, step by step" invoked chain-of-thought behavior but analogous prompts did not.