Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SkillsBench: Benchmarking how well agent skills work across diverse tasks

arxiv.org

Post reply on HN
Print view View original

151–160 of 182 posts

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#151

Post by embedding-shape »

stitched2gethr wrote: post #118
embedding-shape wrote: post #2

The general rule seems to be, the more layers you automate with LLMs, the worse each successive layer gets. Piping LLM output as input into new LLM calls, you're already starting to notice how things fall apart and get lost quickly. If you have the idea, more or less the implementation plan, let the LLM do the coding, you can end up with something maintainable and nice, it's basically up to you. Strip away one layer,…

It's all about how full the context is, right? For a task that can be completed in 20% of the context it doesn't matter, but you don't want to fill your context with exploration before you do the hard part. I have actually found something close to the opposite. I work on a large codebase and I often use the LLM to generate artifacts before performing the task (for complex tasks). I use a prompt to say "go explore thi…

> It's all about how full the context is, right?

No, even when you restart the context from scratch, which I do for each change really, seeing that same effect.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#152

Post by crashabr »

8note wrote: post #117
rapind wrote: post #103

Earlier quoted context omitted.

CLAUDE.md in my projects only has coding / architecture guidelines. Here's what not to do. Here's what you should do. Here are my preferences. Here's where the important things are. Even though my CLAUDE.md is small though, often my rules are ignored. Not always though, so it's still at least somewhat useful!

im trying out some other cc features, and om thinking maybe hooks can do something with this. have a hook on switching out of plan, and maybe on edits, that passes the change to haiku with the claude.md to see if it matches or not

What's the hook for switching out of plan? I'd like to be launch a planning skill whenever claude writes a plan but it never picks up the skill, and I haven't found a hook that can force it to.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#153

Post by Arifcodes »

bavarianbob wrote: post #133
Arifcodes wrote: post #132

[dead]

Do you have a working example of a skill reducing tokens on repeat tasks? I'm personally seeing the cost of writing and maintaining skills to be much larger than the tokens I'm saving by doing so.

[dead]

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#154

Post by btown »

ljm wrote: post #44
btown wrote: post #30

Earlier quoted context omitted.

It's even worse than this: the "tasks" that are evaluated are limited to a single markdown file of instructions, plus an opaque verifier (page 13-14). No problems involving existing codebases, refactors, or anything of the like, where the key constraint is that the "problem definition" in the broadest sense doesn't fit in context. So when we look at the prompt they gave to have the agent generate its own skills: > Im…

I don't see how "create an abstraction before attempting to solve the problem" will ever work as a decent prompt when you are not even steering it towards specifics. If you gave this exact prompt to a senior engineer I would expect them to throw it back and ask wtf you actually want. LLMs are not mind readers.

If it were in the context of parachuting into a codebase, I’d make these skills an important familiarization exercise: how are tests made, what are patterns I see frequently, what are the most important user flows. By forcing myself to distill that first, I’d be better at writing code that is in keeping with the codebase’s style and overarching/subtle goals. But this makes zero sense in a green-field task.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#155

Post by sebastianconcpt »

jibal wrote: post #84
sebastianconcpt wrote: post #80

I am the only one surprised about anyone's need for a study to conclude that?

https://news.ycombinator.com/newsguidelines.html > Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.

Ok fair, thanks for explaining

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#156

Post by JamesSwift »

x3n0ph3n3 wrote: post #124
JamesSwift wrote: post #51

Earlier quoted context omitted.

Yeah some of my most useful AI tooling are skills created via a “role play session”. Basically brain dumping to the agent and telling it to ask questions and figure out how to accomplish a task, then distilling it into a skill at the end which is much tighter and evidence based from the actual problem solving session

This was very insightful. I've only just begun playing with some agent workflows and building out documentation to help it navigate my code base. Asking it to give me the top 10 unanswered questions from analyzing the docs and code was very useful.

YAGNI is the best tool in your toolbox for AI agents. Dont build out what you think will be useful, layer things into your AI toolbox as they prove they are needed. Especially for claude, running `/init` ends up with a lot of really unnecessary/hallucinated info. Keep it all simple and layer on top.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#157

Post by mathisd »

I started building an agentic-ready data warehouse (GitHub.com/mathisdrn/orca) and was thinking that my skills could be optimized by benchmarking them. Turns out there is a better way of optimizing and building them using model languages themselves as evaluator and skill builder. See DsPy and GEPA. I am wondering whether Anthropic and OpenAI skill-creator skill is themselves optimize to optimize skills efficiency on various tasks.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#158

Post by crimsoneer »

siva7 wrote: post #40
j45 wrote: post #36

Earlier quoted context omitted.

Sincerely, perhaps you should publish on arxiv before a researcher reads it to run it and write a study. It's fairly common we notice these types of threads where one thing is being postulated and then there's comments upon comments of doer's showing what they have done.

somehow sad that some random dude on hn seems to have more brain than most scientists publishing on something about agents or prompting.

Scientists are just random dudes (though some of them have been given a fancy hat by a university, which is a recognised mark of accomplishment).

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#159

Post by dang »

mustaphah wrote: post #147
dang wrote: post #141

Submitted title was "Study: Self-generated agent skills are useless". That's against the site guideline: " Please use the original title, unless it is misleading or linkbait; don't editorialize. " - https://news.ycombinator.com/newsguidelines.html If you want to say what you think is important about an article, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field w…

What if the most interesting finding ends up buried under a vague title? Aside from the "self-generated skills" aspect, there isn't much there that meaningfully warrants deeper discussion. I chose a title that directly reflects an interesting finding - something that offers substantial insight to the community. I think the rule should be applied with some nuance; in this case, being explicit is a net positive. I have…

Yes, I appreciate that, and yes there is room for nuance. But I think you went too far in this case, meaning that the delta between the article title and the submission title was too large. For example, the word "useless" appears nowhere in the article abstract nor in the article body. That's a big delta.

I was starting to type out a longer explanation but I ran out of time - however, I probably would just be repeating things I've said many times before, for example here: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... - perhaps some of that would be helpful.

You're a fine HN contributor and obviously a genuine user and I hope I didn't come across as critical! From our side it's just standard HN moderation practice. The way we deal with titles has been stable for many years. It isn't entirely mechanical, there are many subtleties (back to the nuance thing) but the core rules have served the site really well. THe main thing we want to avoid is having the title field be a mini-genre where whoever makes the submission gets to put their spin on the article.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#160

Post by ljm »

btown wrote: post #154
ljm wrote: post #44

Earlier quoted context omitted.

I don't see how "create an abstraction before attempting to solve the problem" will ever work as a decent prompt when you are not even steering it towards specifics. If you gave this exact prompt to a senior engineer I would expect them to throw it back and ask wtf you actually want. LLMs are not mind readers.

If it were in the context of parachuting into a codebase, I’d make these skills an important familiarization exercise: how are tests made, what are patterns I see frequently, what are the most important user flows. By forcing myself to distill that first, I’d be better at writing code that is in keeping with the codebase’s style and overarching/subtle goals. But this makes zero sense in a green-field task.

There's overlap in that with brownfield or legacy code you are strongly opinionated on the status quo, and on the greenfield you are strongly opinionated with fewer constraints.

You have to work with conviction though. It's when you offload everything to the LLM that things start to drift from expectations, because you kept the expectations in your head and away from the prompt.

Post reply on HN