Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SkillsBench: Benchmarking how well agent skills work across diverse tasks

arxiv.org

Post reply on HN
Print view View original

61–70 of 182 posts

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#61

Post by JumpCrisscross »

dcre wrote: post #8

"Self-Generated Skills: No Skills provided, but the agent is prompted to generate relevant procedural knowledge before solving the task. This isolates the impact of LLMs’ latent domain knowledge" This is a useful result, but it is important to note that this is not necessarily what people have in mind when they think of "LLMs generating skills." Having the LLM write down a skill representing the lessons from the stru…

> it is important to note that this is not necessarily what people have in mind when they think of "LLMs generating skills

I’m reading this paper as don’t do this. If you deploy agents to your workforce and tell them to use skills, don’t. Tell them to give it tasks. This sounds obvious but might not be to everyone. (And in any case, it’s nice for researchers to have confirmed pre-prompt skill writing doesn’t work. It would have been neat if it had.)

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#62

Post by verdverm »

small_model wrote: post #54
verdverm wrote: post #48

Earlier quoted context omitted.

That later "learning" part is updating weights in your brain What Ai's get is a cheat sheet for the session

That's what I mean by continual learning, skills, memory are a crutch until real learning can happen, which could be weights changing in the local instance.

And my point is that weight changes are not likely to have the economic ROI for their justification on a person-by-person basis

What you are suggesting is a very expensive late-training phase activity. It's also not clear anymore when fine-tuning helps or hurts. Progress is rapid

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#63

Post by zahlman »

rahimnathwani wrote: post #31

This is unsurprising and irrelevant. When you create a skill for a particular model, you don't typically ask the model to create the skill based solely on its own latent knowledge. Otherwise, you'd expect the effect to be similar to telling the model 'make a plan before acting, make not mistakes'. But that's what the paper's authors did! When they say 'self-generated' they don't allow the model any tool access at all…

> Otherwise, you'd expect the effect to be similar to telling the model 'make a plan before acting, make not mistakes'.

Have there not been previous iterations of these tools where such techniques were actually effective?

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#64

Post by rahimnathwani »

zahlman wrote: post #63
rahimnathwani wrote: post #31

This is unsurprising and irrelevant. When you create a skill for a particular model, you don't typically ask the model to create the skill based solely on its own latent knowledge. Otherwise, you'd expect the effect to be similar to telling the model 'make a plan before acting, make not mistakes'. But that's what the paper's authors did! When they say 'self-generated' they don't allow the model any tool access at all…

> Otherwise, you'd expect the effect to be similar to telling the model 'make a plan before acting, make not mistakes'. Have there not been previous iterations of these tools where such techniques were actually effective?

Yes, but this paper studied recent models.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#66

Post by dimitri-vs »

colonCapitalDee wrote: post #34

I have a custom skill-creator skill that contains this: > A common pitfall is for Claude to create skills and fill them up with generated information about how to complete a task. The problem with this is that the generated content is all content that's already inside Claude's probability space. Claude is effectively telling itself information that it already knows! > Instead, Claude should strive to document in SKIL…

I don't think LLMs are very good at introspection on what they know or don't know, but otherwise this is gold. Thanks for sharing.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#68

Post by small_model »

verdverm wrote: post #62
small_model wrote: post #54

Earlier quoted context omitted.

That's what I mean by continual learning, skills, memory are a crutch until real learning can happen, which could be weights changing in the local instance.

And my point is that weight changes are not likely to have the economic ROI for their justification on a person-by-person basis What you are suggesting is a very expensive late-training phase activity. It's also not clear anymore when fine-tuning helps or hurts. Progress is rapid

I see, I misunderstood your original message. Given how much progress has been made without it, It's perhaps not necessary especially if the economics make it prohibitive.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#69

Post by myhf »

rcarmo wrote: post #56

I only generate skills _after_ I've worked through a problem with the model - usually by asking it "what have you learned in this session?". I have no idea why people would think it can zero-shot a problem space without any guidance or actual experience...

Why would you expect it to generate more effective skills when you aren't even making a salt circle or lighting incense?

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#70

Post by pizza »

The more general question of how to evaluate the quality of a given skill file is quite interesting to me. A skill may prime a model's responses in a way that a prompt alone may not. But also models aren't good at judging what they are or are not capable of.

Just asking a model "how good is this skill?" may or may not work, possibly the next laziest thing you could do - that's still "for cheap" - is asking the model to make a quiz for itself, and have it take the quiz with and without access to the skill, then see how the skill improved it. But there's still many problems with that approach. But would it be useful enough to work well enough much of the time for just heuristically estimating the quality of a skill?

Post reply on HN