Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SkillsBench: Benchmarking how well agent skills work across diverse tasks

arxiv.org

Post reply on HN
Print view View original

51–60 of 182 posts

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#51

Post by JamesSwift »

dcre wrote: post #8

"Self-Generated Skills: No Skills provided, but the agent is prompted to generate relevant procedural knowledge before solving the task. This isolates the impact of LLMs’ latent domain knowledge" This is a useful result, but it is important to note that this is not necessarily what people have in mind when they think of "LLMs generating skills." Having the LLM write down a skill representing the lessons from the stru…

Yeah some of my most useful AI tooling are skills created via a “role play session”. Basically brain dumping to the agent and telling it to ask questions and figure out how to accomplish a task, then distilling it into a skill at the end which is much tighter and evidence based from the actual problem solving session

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#52

Post by secbear »

The finding that self-generated skills provide negative benefit (-1.3pp) while curated skills give +16.2pp is the most interesting result here imo. Big discrepancy, but makes sense. Aligns with the thought that LLMs are better consumers of procedural knowledge than producers of it.

+4.5pp for software engineering is suspiciously low compared to +51.9pp for healthcare. I suspect this reflects that frontier models already have strong SWE priors from training data, so skills add less marginal value. If true, skills become most valuable precisely in the domains where models are weakest — which is where you'd actually want to deploy agents in production. That's encouraging.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#53

Post by zozbot234 »

deadbabe wrote: post #39
zozbot234 wrote: post #37

Earlier quoted context omitted.

The point of so-called 'skills' is to be short how-to reminders that the agent can pull into its context and then act upon. If the knowledge is already in the model, it will most likely be surfaced in reasoning phase anyway, so there's little benefit to writing it up as a skill, unless perhaps it's extremely relevant and hard to surface, and you want the model to skip that part of the reasoning.

There is a benefit of a skill though. If an AI keeps encoding common tasks as skills and scripts, the LLM eventually just becomes a dumb routing mechanism for ambiguous user requests, which ultimately drives down token usage. If everything you want an LLM do is already captured as code or simple skills, you can switch to dumber models which know enough about selecting the appropriate skill for a given user input, and…

AI companies don't want you to waste tokens, they benefit when you use them efficiently because they can serve more users on the infra that's the main bottleneck for them. It's Jevons' paradox in action.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#54

Post by small_model »

verdverm wrote: post #48
small_model wrote: post #27

Earlier quoted context omitted.

I guess when I was in class and took notes, then reviewed them later I wasn't "learning" anything.

That later "learning" part is updating weights in your brain What Ai's get is a cheat sheet for the session

That's what I mean by continual learning, skills, memory are a crutch until real learning can happen, which could be weights changing in the local instance.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#55

Post by hardware2415 »

secbear wrote: post #52

The finding that self-generated skills provide negative benefit (-1.3pp) while curated skills give +16.2pp is the most interesting result here imo. Big discrepancy, but makes sense. Aligns with the thought that LLMs are better consumers of procedural knowledge than producers of it. +4.5pp for software engineering is suspiciously low compared to +51.9pp for healthcare. I suspect this reflects that frontier models alre…

[flagged]

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#56

Post by rcarmo »

I only generate skills _after_ I've worked through a problem with the model - usually by asking it "what have you learned in this session?". I have no idea why people would think it can zero-shot a problem space without any guidance or actual experience...

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#57

Post by nimonian »

embedding-shape wrote: post #2

The general rule seems to be, the more layers you automate with LLMs, the worse each successive layer gets. Piping LLM output as input into new LLM calls, you're already starting to notice how things fall apart and get lost quickly. If you have the idea, more or less the implementation plan, let the LLM do the coding, you can end up with something maintainable and nice, it's basically up to you. Strip away one layer,…

It's like those sequences of images where we ask the LLM to reproduce the same image exactly, and we just get some kind of grotesque collapse after a few dozen iterations. The same happens with text and code. I call this "semantic collapse".

I conjecture that after some years of LLMs reading a SharePoint site, producing summaries, then summaries of those summaries, etc... We will end up with a grotesque slurry.

At some point, fresh human input is needed to inject something meaningful into the process.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#58

Post by nvader »

hardware2415 wrote: post #55
secbear wrote: post #52

The finding that self-generated skills provide negative benefit (-1.3pp) while curated skills give +16.2pp is the most interesting result here imo. Big discrepancy, but makes sense. Aligns with the thought that LLMs are better consumers of procedural knowledge than producers of it. +4.5pp for software engineering is suspiciously low compared to +51.9pp for healthcare. I suspect this reflects that frontier models alre…

[flagged]

Hmm, not for me, but I'm curious if there are signatures I'm missing.

To me, author reads like an articulate native English speaker, but typing on their phone.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#59

Post by nvader »

Exoristos wrote: post #49
lukev wrote: post #46

This clarifies an important point for me. The derivative of a LLM agent's capabilities (on its own) is negative. It's not that they can't do useful work -- it means that (for now) they require some level of input or steering. If that were to change -- if an agent could consistently get better at what it does without intervention -- that would represent a true paradigm shift. An accelerating curve, rather than one tre…

Just to save us all some time and trouble, I'll point out that that's never really going to happen.

Source?

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#60

Post by [deleted] »

hardware2415 wrote: post #55
secbear wrote: post #52

The finding that self-generated skills provide negative benefit (-1.3pp) while curated skills give +16.2pp is the most interesting result here imo. Big discrepancy, but makes sense. Aligns with the thought that LLMs are better consumers of procedural knowledge than producers of it. +4.5pp for software engineering is suspiciously low compared to +51.9pp for healthcare. I suspect this reflects that frontier models alre…

[flagged]

[deleted]
Post reply on HN