Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SkillsBench: Benchmarking how well agent skills work across diverse tasks

arxiv.org

Post reply on HN
Print view View original

131–140 of 182 posts

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#131

Post by allthetime »

I can't help but feel that far too many intelligent people, including many here, are wasting too much of their precious time, skill, potential, etc. on questions like this. Remember a few years ago when we just used to make useful software? Now we are consumed with discussions about the AI flavour of the week and trying really hard to prove the usefulness of things that we will soon forget when the next shiny one comes.

Web3 and JavaScript frameworks never had the nerd-sniping power of the AI ecosystem. I'm not denying the usefulness and potential of the space, and the achievements of its current champions, but the degree with which it has consumed discussion and productivity in the tech space is worrying.

This article would be wildly interesting with the opposite headline, but instead it simply states what many of us would assume based on experience.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#134

Post by yieldcrv »

allthetime wrote: post #131

I can't help but feel that far too many intelligent people, including many here, are wasting too much of their precious time, skill, potential, etc. on questions like this. Remember a few years ago when we just used to make useful software? Now we are consumed with discussions about the AI flavour of the week and trying really hard to prove the usefulness of things that we will soon forget when the next shiny one com…

it's a distributed evolution occurring right now, and lots of people replicate the same things, so its useful to be able to point to some things as a waste of time

that being said, I think you're right that all of this will be a moot point in like 2 weeks or 2 months, when the next AI model is released that addresses this specific friction

and yeah, that's sad. there are a lot of people in companies being instructed to pivot to skills, and then before they can launch or sell their procedurally generated moat, the next AI model will procedurally generate skills better

nobody knows what to do for guaranteed food and shelter so they're grasping

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#135

Post by JB_5000 »

Interesting benchmark, but worth noting the methodology: skills are generated before the task, with no feedback loop. In practice, useful skills tend to emerge from doing — you attempt, observe what failed, then codify what worked. Generate → execute → observe → refine. The paper tests cold generation, which is a different (and less realistic) setup.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#136

Post by BoredomIsFun »

getoffit wrote: post #41

"Small models" will always outperform as they are deterministic (or closer to it). This was realized in 2023 already: https://newsletter.semianalysis.com/p/google-we-have-no-moat... "Less is best" is not a new realization. The concept exists across contexts. Music described as "overplayed". Prose described as verbose. We just went through an era of compute that chanted "break down your monoliths". NPM ecosystem being…

> "Small models" will always outperform as they are deterministic (or closer to it).

Your whole comment feels like, pardon me, like LARPing. No, small models do not outperform the large ones, unless finetuned. Saying that as someone who uses small models 95% vs cloud ones.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#138

Post by eigenblake »

Biggest limitation I see in this paper: the framing. Any time you have a lot of proprietary knowledge or you've just sorted out the right solution when it's not readily available from the model's parametric knowledge, that's when you should add a skill. Wrap it in a CLI that's easy to inspect. You don't need to store the whole help text of the skill either. The model can inspect it and its subcommands.

Reality doesn't force us to choose between skill or no skill, reality often doesn't give us a choice. You can either make a skill for your company's proprietary system or your model has to figure it out from scratch every time by searching wikis or reading code. If you use it right, skills are a compression mechanism. Instead of the process meaning your model needs to get all of theses files dynamically, it can simply statically run.

To steel-man the paper. It is worth looking at whether you should try to code something up first or try a skill first. And it may well be valid to say try first and if you can't work it out in 5 mins, install a skill. But there's a meta point of skills as software (where you deduplicate the effort of solving regressions).

For a reductio ad absurdum, If self-generated skills with no additional context _didn't_ eventually level off in performance, then we could reach AGI by making one big skill that keeps growing and solving harder and harder tasks, including improving the capability of its own skill-builder skill, all without embedding any signals from the environment or needing to interface with the real world at all.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#139

Post by st-msl »

What's interesting to me isn't the self-generated finding (everyone here has correctly identified the methodology issue). It's Table 4 buried on page 6.

Healthcare +51.9pp. Manufacturing +41.9pp. Software Engineering +4.5pp.

The domains where models have the weakest priors from pretraining benefit the most from external procedural knowledge. That's not surprising on its own, but there's an implication I haven't seen anyone raise: these are exactly the enterprise domains where that procedural knowledge is most proprietary and most dangerous to lose between sessions.

The paper's entire architecture is single-player. A SKILL.md sits in a directory, one agent reads it, session ends. When Agent A at a bank figures out the right approach to parsing 13F filings (0% to 75% with the right skill in this paper), that knowledge dies with the context window. Agent B starts from scratch.

We're building shared memory infrastructure for agents at Memco (https://memco.ai) and this paper maps directly to what our enterprise design partners keep telling us — the problem isn't writing skills, it's that procedural knowledge doesn't compound across agents, sessions, or teams. The paper even shows 2-3 focused skills outperform comprehensive docs, which is a retrieval problem masquerading as an authoring problem.

The question this paper should be asking isn't "can agents write their own skills" — it's "what infrastructure makes skills accumulate and transfer?" Static files in a folder is the wrong primitive for that.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#140

Post by D-Machine »

Insofar as they are testing very broken and simple "skills" (as evidenced by comments in this thread), and still find net significant net positives in some cases, negatives in others, and huge variability overall, I actually think this is a fun paper providing support that, at least if you count using skills here, you can, in general, very objectively and quantifiably be holding it right/wrong.

I.e. your use of skills resulting in differences of up to ~52 percentage points (or negative percentage points) in improvements (or degradations) in your percentage pass rate is a decent first-pass measure of the importance of skills here.

Post reply on HN