Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SkillsBench: Benchmarking how well agent skills work across diverse tasks

arxiv.org

Post reply on HN
Print view View original

91–100 of 182 posts

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#91

Post by nguyentran03 »

getoffit wrote: post #41

"Small models" will always outperform as they are deterministic (or closer to it). This was realized in 2023 already: https://newsletter.semianalysis.com/p/google-we-have-no-moat... "Less is best" is not a new realization. The concept exists across contexts. Music described as "overplayed". Prose described as verbose. We just went through an era of compute that chanted "break down your monoliths". NPM ecosystem being…

Small models aren't more deterministic than large ones. Determinism comes from temperature and sampling settings, not parameter count. A 7B model at temp 0.7 is just as stochastic as a 405B model.

The "no moat" memo you linked was about open source catching up to closed models through fine-tuning, not about small models outperforming large ones.

I'm also not sure what "skeletonized down to opcodes" or "geometry for text as bytecode patterns" means in the context of neural networks. Model compression is a real field (quantization, distillation, pruning) but none of it works the way you're describing here.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#92

Post by nmilo »

colonCapitalDee wrote: post #34

I have a custom skill-creator skill that contains this: > A common pitfall is for Claude to create skills and fill them up with generated information about how to complete a task. The problem with this is that the generated content is all content that's already inside Claude's probability space. Claude is effectively telling itself information that it already knows! > Instead, Claude should strive to document in SKIL…

This is really good! I like how it reads like a blog post, it feels like I'm learning a skill on how to write good skills. Maybe that's another heuristic, a skill should read like an interesting blog post, highlighting non-obvious information.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#93

Post by mhmmmmmm »

zozbot234 wrote: post #53
deadbabe wrote: post #39

Earlier quoted context omitted.

There is a benefit of a skill though. If an AI keeps encoding common tasks as skills and scripts, the LLM eventually just becomes a dumb routing mechanism for ambiguous user requests, which ultimately drives down token usage. If everything you want an LLM do is already captured as code or simple skills, you can switch to dumber models which know enough about selecting the appropriate skill for a given user input, and…

AI companies don't want you to waste tokens, they benefit when you use them efficiently because they can serve more users on the infra that's the main bottleneck for them. It's Jevons' paradox in action.

I don't think thats necessarily true, they aren't really capacity constrained in practice (they might be behind the scenes and adjust training on the fly, but thats speculation), so wasting tokens effectively helps utilize their (potentially idle) inference GPU's

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#94

Post by cheema33 »

secbear wrote: post #52

The finding that self-generated skills provide negative benefit (-1.3pp) while curated skills give +16.2pp is the most interesting result here imo. Big discrepancy, but makes sense. Aligns with the thought that LLMs are better consumers of procedural knowledge than producers of it. +4.5pp for software engineering is suspiciously low compared to +51.9pp for healthcare. I suspect this reflects that frontier models alre…

> +4.5pp for software engineering is suspiciously low compared to +51.9pp for healthcare.

This stood out for me as well. I do think that LLMs have a lot of training data on software engineering topics and that perhaps explains the large discrepancy. My experience has been that if I am working with a software library or tool that is very new or not commonly used, skills really shine there. Example: Adobe React Spectrum UI library. Without skills, Opus 4.6 produces utter garbage when trying to use this library. With properly curated/created skills, it shines. Massive difference.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#95

Post by gruez »

zozbot234 wrote: post #53
deadbabe wrote: post #39

Earlier quoted context omitted.

There is a benefit of a skill though. If an AI keeps encoding common tasks as skills and scripts, the LLM eventually just becomes a dumb routing mechanism for ambiguous user requests, which ultimately drives down token usage. If everything you want an LLM do is already captured as code or simple skills, you can switch to dumber models which know enough about selecting the appropriate skill for a given user input, and…

AI companies don't want you to waste tokens, they benefit when you use them efficiently because they can serve more users on the infra that's the main bottleneck for them. It's Jevons' paradox in action.

>AI companies don't want you to waste tokens, they benefit when you use them efficiently because they can serve more users on the infra that's the main bottleneck for them.

No, the actual incentive is that people will eventually benchmark their models on bang-per-buck basis and models that chew through tokens are not going to be competitive. It's the same reason why the "Intel/AMD are intentionally sandbagging their CPUs so they can sell more CPUs" theory doesn't work.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#96

Post by achille »

smcleod wrote: post #12

There is almost no point in telling an agent to build a skill without augmenting it's knowledge on the thing it's writing about as you're just piping output to input without expanding the information in the system. If you get an agent to perform a bunch of research online, distil that down to information that the models tend not to get right or is newer than what is in their training data or simply better aligns with…

Absolutely, they didn't give the agents autonomy to research or any additional data. No documentation, no web search, no reference materials.

What's the point of building skills like this?

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#97

Post by jibal »

lukev wrote: post #46

This clarifies an important point for me. The derivative of a LLM agent's capabilities (on its own) is negative. It's not that they can't do useful work -- it means that (for now) they require some level of input or steering. If that were to change -- if an agent could consistently get better at what it does without intervention -- that would represent a true paradigm shift. An accelerating curve, rather than one tre…

Doesn't anyone learn from Malthus? In the real world, accelerating curves inevitably stop accelerating.

Others here have suggested that AIs should be able to self-generate skills by doing web searches. What happens when all of the information from web searches (of knowledge generated by ordinary human intelligence) has been extracted?

On another post (about crackpot Nick Bostrom claiming that an ASI would "imminently" lead to scientific breakthroughs like curing Alzheimers and so a 3% chance of developing an ASI would be worth a 97% chance of annihilating humanity) I noted that an ASI isn't a genie or magic wand; it can't find the greatest prime or solve the halting problem. Another person noted that an ASI can't figure out how to do a linear search in O(1) time. (We already know how to do a table lookup in amortized O(1) time--build a hash table.) Science is like animal breeding and many other processes ... there's a limit to how much it can be sped up.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#98

Post by cheema33 »

rahimnathwani wrote: post #31

This is unsurprising and irrelevant. When you create a skill for a particular model, you don't typically ask the model to create the skill based solely on its own latent knowledge. Otherwise, you'd expect the effect to be similar to telling the model 'make a plan before acting, make not mistakes'. But that's what the paper's authors did! When they say 'self-generated' they don't allow the model any tool access at all…

> This is unsurprising and irrelevant. When you create a skill for a particular model, you don't typically ask the model to create the skill based solely on its own latent knowledge.

This!

The only surprising part about the paper is that somebody wrote a paper on skills without a good understanding of the topic.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#100

Post by cheema33 »

rcarmo wrote: post #56

I only generate skills _after_ I've worked through a problem with the model - usually by asking it "what have you learned in this session?". I have no idea why people would think it can zero-shot a problem space without any guidance or actual experience...

> I only generate skills _after_ I've worked through a problem with the model.

This is the correct way vast majority of the time. There are exceptions. When I know for certain that the models do not have enough training material on a new library or one that isn't often used or an internal tool. In those cases I know I will have struggle on my hand if I don't start out with a skill that teaches the model the basics of what it does not know. I then update the skill with more polish as we discover additional ways it can be improved. Any errors the model makes are used to improve existing skills or create new ones.

Post reply on HN