Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SkillsBench: Benchmarking how well agent skills work across diverse tasks

arxiv.org

Post reply on HN
Print view View original

121–130 of 182 posts

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#121

Post by 7thpower »

rapind wrote: post #103
jwpapi wrote: post #89

Earlier quoted context omitted.

Thats actually super interesting and why I really don’t like the whole .md folder structures or even any CLAUDE.md. It just seems most of the time you really just want to give it what it needs for best results. The headline is really bullshit, yes, I like the testing tho.

CLAUDE.md in my projects only has coding / architecture guidelines. Here's what not to do. Here's what you should do. Here are my preferences. Here's where the important things are. Even though my CLAUDE.md is small though, often my rules are ignored. Not always though, so it's still at least somewhat useful!

I’m pretty sure Claude just uses mine to keep a running list of pressure points for when I get cross with it.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#122

Post by 7thpower »

awwaiid wrote: post #109
zozbot234 wrote: post #37

Earlier quoted context omitted.

The point of so-called 'skills' is to be short how-to reminders that the agent can pull into its context and then act upon. If the knowledge is already in the model, it will most likely be surfaced in reasoning phase anyway, so there's little benefit to writing it up as a skill, unless perhaps it's extremely relevant and hard to surface, and you want the model to skip that part of the reasoning.

I've been building a skill to help run manual tests on an app. So I go through and interactively steer toward a useful validation of a particular PR, navigating specifics of the app and what I care about and what I don't. Then in the end I have it build a skill that would have skipped backtracking and retries and the steering I did. Then I do it again from scratch; this time it takes less steering. I have it update t…

I love the screenshots, I need to do something like that.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#123

Post by dalemhurley »

dcre wrote: post #8

"Self-Generated Skills: No Skills provided, but the agent is prompted to generate relevant procedural knowledge before solving the task. This isolates the impact of LLMs’ latent domain knowledge" This is a useful result, but it is important to note that this is not necessarily what people have in mind when they think of "LLMs generating skills." Having the LLM write down a skill representing the lessons from the stru…

After several failures then a success I have the agent create the skill, next run it is successful first run.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#124

Post by x3n0ph3n3 »

JamesSwift wrote: post #51
dcre wrote: post #8

"Self-Generated Skills: No Skills provided, but the agent is prompted to generate relevant procedural knowledge before solving the task. This isolates the impact of LLMs’ latent domain knowledge" This is a useful result, but it is important to note that this is not necessarily what people have in mind when they think of "LLMs generating skills." Having the LLM write down a skill representing the lessons from the stru…

Yeah some of my most useful AI tooling are skills created via a “role play session”. Basically brain dumping to the agent and telling it to ask questions and figure out how to accomplish a task, then distilling it into a skill at the end which is much tighter and evidence based from the actual problem solving session

This was very insightful. I've only just begun playing with some agent workflows and building out documentation to help it navigate my code base. Asking it to give me the top 10 unanswered questions from analyzing the docs and code was very useful.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#125

Post by dalemhurley »

isahers wrote: post #24
dcre wrote: post #8

"Self-Generated Skills: No Skills provided, but the agent is prompted to generate relevant procedural knowledge before solving the task. This isolates the impact of LLMs’ latent domain knowledge" This is a useful result, but it is important to note that this is not necessarily what people have in mind when they think of "LLMs generating skills." Having the LLM write down a skill representing the lessons from the stru…

Yeah I care about LLM's generating skills after attempting tasks and learning lessons from those attempts, not before attempting a task for the first time. This result seems a little silly and detached from the reality of how skills are "auto-generated" in the real world.

That is my approach. I don’t think the papers author has actually used skills.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#126

Post by therealdrag0 »

rahimnathwani wrote: post #31

This is unsurprising and irrelevant. When you create a skill for a particular model, you don't typically ask the model to create the skill based solely on its own latent knowledge. Otherwise, you'd expect the effect to be similar to telling the model 'make a plan before acting, make not mistakes'. But that's what the paper's authors did! When they say 'self-generated' they don't allow the model any tool access at all…

Modern science encourages publishing non-surprising results.

And also I’ve seen my manager LARP as an engineer by asking a model to generate a best practices doc for a service repo without supplying any additional context. So this sort of paper helps discourage that behavior.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#128

Post by zahlman »

gwern wrote: post #102
zahlman wrote: post #63

Earlier quoted context omitted.

> Otherwise, you'd expect the effect to be similar to telling the model 'make a plan before acting, make not mistakes'. Have there not been previous iterations of these tools where such techniques were actually effective?

But that's a reason you should expect it to stop working soon, just like all the older tricks like "my grandmother will die". If you have a universal 'blind' prompt which can increase performance a little bit... the AI labs can just toss that into the training loop to teach the model to do it automatically, whatever 'it' was, like 'trying harder' or 'writing down a useful idea'. And then the prompt stops working beca…

> But that's a reason you should expect it to stop working soon

I agree. (And it seems like it already stopped working, if I understood others here correctly.)

But again if I understood others here correctly, an academic paper like this would necessarily be studying models that are well behind the leading edge at time of publication. My argument is that the study authors shouldn't be faulted for investigating something that currently seems unlikely to work, because at the time of investigation it would have seemed much more likely to work.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#129

Post by balls187 »

ljm wrote: post #44
btown wrote: post #30

Earlier quoted context omitted.

It's even worse than this: the "tasks" that are evaluated are limited to a single markdown file of instructions, plus an opaque verifier (page 13-14). No problems involving existing codebases, refactors, or anything of the like, where the key constraint is that the "problem definition" in the broadest sense doesn't fit in context. So when we look at the prompt they gave to have the agent generate its own skills: > Im…

I don't see how "create an abstraction before attempting to solve the problem" will ever work as a decent prompt when you are not even steering it towards specifics. If you gave this exact prompt to a senior engineer I would expect them to throw it back and ask wtf you actually want. LLMs are not mind readers.

Interesting.

I think it's because AI Models have learned that we prefer answers that are confident sounding, and not to pester us with questions before giving us an answer.

That is, follow my prompt, and don't bother me about it.

Because if I am coming to an AI Agent to do something, it's because I'd rather be doing something else.

Post reply on HN