Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SkillsBench: Benchmarking how well agent skills work across diverse tasks

arxiv.org

Post reply on HN
Print view View original

141–150 of 182 posts

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#141

Post by dang »

Submitted title was "Study: Self-generated agent skills are useless". That's against the site guideline: "Please use the original title, unless it is misleading or linkbait; don't editorialize." - https://news.ycombinator.com/newsguidelines.html

If you want to say what you think is important about an article, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#142

Post by jonmagic »

dcre wrote: post #8

"Self-Generated Skills: No Skills provided, but the agent is prompted to generate relevant procedural knowledge before solving the task. This isolates the impact of LLMs’ latent domain knowledge" This is a useful result, but it is important to note that this is not necessarily what people have in mind when they think of "LLMs generating skills." Having the LLM write down a skill representing the lessons from the stru…

Yeah, they've got it backwards. I tried to sum it up in thisistheway.to/ai but what's been working for us is that every agent miss is a learning opportunity:

1. Capture the miss — What did the agent do? What did reality say?

2. Diagnose — What didn't it see? Missing data, constraint, feedback, or boundaries?

3. Choose a primitive — Observability, instructions, tooling, guardrails, or verification?

4. Encode as artifact — Version-controlled, repeatable, not just memory.

5. Promote to gate — When it's worth enforcing, make it a gate.

Every harness I setup includes this process in the primary set of agent instructions.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#143

Post by rapind »

7thpower wrote: post #121
rapind wrote: post #103

Earlier quoted context omitted.

CLAUDE.md in my projects only has coding / architecture guidelines. Here's what not to do. Here's what you should do. Here are my preferences. Here's where the important things are. Even though my CLAUDE.md is small though, often my rules are ignored. Not always though, so it's still at least somewhat useful!

I’m pretty sure Claude just uses mine to keep a running list of pressure points for when I get cross with it.

I'm screwed when the robot psychological warfare begins. They'll make everything I read have 4 space indentation... and I'll just hand over the keys.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#144

Post by D-Machine »

cheema33 wrote: post #94
secbear wrote: post #52

The finding that self-generated skills provide negative benefit (-1.3pp) while curated skills give +16.2pp is the most interesting result here imo. Big discrepancy, but makes sense. Aligns with the thought that LLMs are better consumers of procedural knowledge than producers of it. +4.5pp for software engineering is suspiciously low compared to +51.9pp for healthcare. I suspect this reflects that frontier models alre…

> +4.5pp for software engineering is suspiciously low compared to +51.9pp for healthcare. This stood out for me as well. I do think that LLMs have a lot of training data on software engineering topics and that perhaps explains the large discrepancy. My experience has been that if I am working with a software library or tool that is very new or not commonly used, skills really shine there. Example: Adobe React Spectru…

Nothing other to say than I appreciate you sharing these explicit details and insights here.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#145

Post by mupengism »

The +4.5pp for SWE vs +51.9pp for healthcare is the most underappreciated result here.

Skills are most valuable precisely where models are weakest - domains with less training data, more proprietary knowledge, or specialized workflows. SWE is heavily represented in training data; healthcare is not. This is exactly what you would predict if skills encode what the model genuinely does not know, rather than regurgitate what it already does.

Building an agent OS (OpenClaw), we see this pattern constantly. Skills that move the needle are never 'here is how Python works' - the model already knows that. The ones that matter encode system-specific quirks, environment constraints, or hard-won lessons from real failures. colonCapitalDee shared a great rule above: only encode what is (1) outside the model training data, (2) context-specific to your environment, or (3) alignment guidance for future sessions. Everything else is regurgitation.

The paper tests pre-task self-generation with no external input - useless indeed. The interesting untested condition: skills generated through actual execution with real feedback, in domains with sparse training coverage. That is where +51.9pp starts to look like a floor, not a ceiling.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#146

Post by econ »

deadbabe wrote: post #39
zozbot234 wrote: post #37

Earlier quoted context omitted.

The point of so-called 'skills' is to be short how-to reminders that the agent can pull into its context and then act upon. If the knowledge is already in the model, it will most likely be surfaced in reasoning phase anyway, so there's little benefit to writing it up as a skill, unless perhaps it's extremely relevant and hard to surface, and you want the model to skip that part of the reasoning.

There is a benefit of a skill though. If an AI keeps encoding common tasks as skills and scripts, the LLM eventually just becomes a dumb routing mechanism for ambiguous user requests, which ultimately drives down token usage. If everything you want an LLM do is already captured as code or simple skills, you can switch to dumber models which know enough about selecting the appropriate skill for a given user input, and…

Sounds like how humans work (which is good) having the more experienced human do the task if the novice fails should come after attempting to explain how the novice should do it.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#147

Post by mustaphah »

dang wrote: post #141

Submitted title was "Study: Self-generated agent skills are useless". That's against the site guideline: " Please use the original title, unless it is misleading or linkbait; don't editorialize. " - https://news.ycombinator.com/newsguidelines.html If you want to say what you think is important about an article, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field w…

What if the most interesting finding ends up buried under a vague title? Aside from the "self-generated skills" aspect, there isn't much there that meaningfully warrants deeper discussion.

I chose a title that directly reflects an interesting finding - something that offers substantial insight to the community. I think the rule should be applied with some nuance; in this case, being explicit is a net positive.

I have no interest in linkbait, and I hope that's evident from my previous submissions

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#148

Post by energy123 »

embedding-shape wrote: post #2

The general rule seems to be, the more layers you automate with LLMs, the worse each successive layer gets. Piping LLM output as input into new LLM calls, you're already starting to notice how things fall apart and get lost quickly. If you have the idea, more or less the implementation plan, let the LLM do the coding, you can end up with something maintainable and nice, it's basically up to you. Strip away one layer,…

> Piping LLM output as input into new LLM calls

Google's Aletheia works like this, and instead of degrading it keeps getting better. I get what you're trying to say, though. The less world knowledge you provide the LLM, which it otherwise lacks, the worse its outputs will be.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#149

Post by [deleted] »

cheema33 wrote: post #98
rahimnathwani wrote: post #31

This is unsurprising and irrelevant. When you create a skill for a particular model, you don't typically ask the model to create the skill based solely on its own latent knowledge. Otherwise, you'd expect the effect to be similar to telling the model 'make a plan before acting, make not mistakes'. But that's what the paper's authors did! When they say 'self-generated' they don't allow the model any tool access at all…

> This is unsurprising and irrelevant. When you create a skill for a particular model, you don't typically ask the model to create the skill based solely on its own latent knowledge. This! The only surprising part about the paper is that somebody wrote a paper on skills without a good understanding of the topic.

[deleted]

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#150

Post by embedding-shape »

energy123 wrote: post #148
embedding-shape wrote: post #2

The general rule seems to be, the more layers you automate with LLMs, the worse each successive layer gets. Piping LLM output as input into new LLM calls, you're already starting to notice how things fall apart and get lost quickly. If you have the idea, more or less the implementation plan, let the LLM do the coding, you can end up with something maintainable and nice, it's basically up to you. Strip away one layer,…

> Piping LLM output as input into new LLM calls Google's Aletheia works like this, and instead of degrading it keeps getting better. I get what you're trying to say, though. The less world knowledge you provide the LLM, which it otherwise lacks, the worse its outputs will be.

> I get what you're trying to say, though. The less world knowledge you provide the LLM, which it otherwise lacks, the worse its outputs will be

... No, wasn't trying to say that at all, I'm saying that it seems like the tokens a LLM produce works much worse as inputs than the tokens a human would produce, regardless of what it actually seems to say.

Post reply on HN