Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SkillsBench: Benchmarking how well agent skills work across diverse tasks

arxiv.org

Post reply on HN
Print view View original

111–120 of 182 posts

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#111

Post by godelski »

sweetjuly wrote: post #14
godelski wrote: post #11

Earlier quoted context omitted.

People like to make the comparison between zip file compressions, where you can degrade something by continually compressing. Same with using jpeg or mp3. But I like to use the analogy of the game "Telephone" (also called "Chinese Whispers"). I think it also highlights how fraught natural language is and just how quickly it can degrade. I think a lot of people are insufficiently impressed with how good we are at comm…

I suggest you find a new DEFLATE library if you're losing data when you compress things with it :)

You do realize there is both lossy and lossless compression, right?

Or did you hyperfixate on the colloquial usage of zip

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#112

Post by CharlieDigital »

dehsge wrote: post #83
CharlieDigital wrote: post #32

Earlier quoted context omitted.

The LLM can be non-deterministic, but in the end, as long as we have compilers and integration tests, isn't it the same? You go from non-deterministic human interpretation of requirements and specs into a compiled, deterministic state machine. Now you have a non-deterministic coding agent doing the same and simply replacing the typing portion of that work. So long as you supply the agent well-curated set of guidance,…

Compilers can never be error free for non trivial statements. This is outlined in Rices theorem. It’s one of the reasons we have observability/telemetry as well as tests.

That's fine, but this also applies to human written code and human written code will have even more variance by skill and experience.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#113

Post by lkoczorowski »

colonCapitalDee wrote: post #34

I have a custom skill-creator skill that contains this: > A common pitfall is for Claude to create skills and fill them up with generated information about how to complete a task. The problem with this is that the generated content is all content that's already inside Claude's probability space. Claude is effectively telling itself information that it already knows! > Instead, Claude should strive to document in SKIL…

Does this not assume that Claude can pick out the best of what it knows?

Claude's training data is the internet. The internet is full of Express tutorials that use app.use(cors()) with no origin restriction. Stack Overflow answers that store JWTs in localStorage, etc.

Claude's probability space isn't a clean hierarchy of "best to worst." It's a weighted distribution shaped by frequency in training data.

So even though it "knows" stuff, it doesn't necessarily know what you want, or what a professional in production environment do.

Unless I'm missing something?

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#114

Post by daxfohl »

daxfohl wrote: post #110

I'm kinda surprised by this. Yeah it's just regurgitating something it already "knows", but I'd still expect that having the skill materialized there in the context would give it something concrete to reference, less likelihood of getting lost or hallucinating, and probably need less incremental context to do the job. I mean, basically it's doing the same thing as reasoning IIUC, except up-front rather than inline an…

Actually, anthropomorphizing a bit, if I take something I vaguely remember how to do, say integration by parts, then if I turn off my own brain's "reasoning", then the "skill" I would generate would almost certainly be wrong, and no help in solving a problem. But if I turn reasoning on, I'd probably be able to come up with the correct algorithm and write it as a skill, sure, but if reasoning is on, I'd be able to solve the problem without needing to write down the algorithm itself (and might even be more successful that way, with something concrete to work with).

OTOH something I know innately how to do, like long division, writing down the algorithm doesn't help at all. In fact if someone just gave me that algorithm and for whatever reason I didn't recognize what it was, I'd have a lot harder time following the instructions than just innately dividing the numbers.

If course anthropomorphizing is always dangerous, but it does provide potential reasons why my above rationale could be wrong.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#116

Post by daxfohl »

I wonder though if self generated skills by advanced models would improve performance of those tasks by small models, or skills created with reasoning mode enabled would help execution of those skills when reasoning is turned off.

For repetitive tasks, that could still be a good way to save on tokens and cost, while still remaining fully automated.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#117

Post by 8note »

rapind wrote: post #103
jwpapi wrote: post #89

Earlier quoted context omitted.

Thats actually super interesting and why I really don’t like the whole .md folder structures or even any CLAUDE.md. It just seems most of the time you really just want to give it what it needs for best results. The headline is really bullshit, yes, I like the testing tho.

CLAUDE.md in my projects only has coding / architecture guidelines. Here's what not to do. Here's what you should do. Here are my preferences. Here's where the important things are. Even though my CLAUDE.md is small though, often my rules are ignored. Not always though, so it's still at least somewhat useful!

im trying out some other cc features, and om thinking maybe hooks can do something with this.

have a hook on switching out of plan, and maybe on edits, that passes the change to haiku with the claude.md to see if it matches or not

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#118

Post by stitched2gethr »

embedding-shape wrote: post #2

The general rule seems to be, the more layers you automate with LLMs, the worse each successive layer gets. Piping LLM output as input into new LLM calls, you're already starting to notice how things fall apart and get lost quickly. If you have the idea, more or less the implementation plan, let the LLM do the coding, you can end up with something maintainable and nice, it's basically up to you. Strip away one layer,…

It's all about how full the context is, right? For a task that can be completed in 20% of the context it doesn't matter, but you don't want to fill your context with exploration before you do the hard part.

I have actually found something close to the opposite. I work on a large codebase and I often use the LLM to generate artifacts before performing the task (for complex tasks). I use a prompt to say "go explore this area if the code and write about it". It documents concepts and has pointers to specific code. Then a fresh session can use that without reading the stuff that doesn't matter. It uses more tokens overall, but includes important details that can get totally missed when you just let it go.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#119

Post by neya »

dcre wrote: post #8

"Self-Generated Skills: No Skills provided, but the agent is prompted to generate relevant procedural knowledge before solving the task. This isolates the impact of LLMs’ latent domain knowledge" This is a useful result, but it is important to note that this is not necessarily what people have in mind when they think of "LLMs generating skills." Having the LLM write down a skill representing the lessons from the stru…

> I'm sure news outlets and popular social media accounts will use appropriate caution in reporting this, and nobody will misunderstand it.

You mean the dude who writes articles on TechCrunch and Ars Technica based off of HN and Reddit thread titles because he doesn't understand what real journalism is? Sure, we can count on him :)

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#120

Post by stitched2gethr »

rahimnathwani wrote: post #31

This is unsurprising and irrelevant. When you create a skill for a particular model, you don't typically ask the model to create the skill based solely on its own latent knowledge. Otherwise, you'd expect the effect to be similar to telling the model 'make a plan before acting, make not mistakes'. But that's what the paper's authors did! When they say 'self-generated' they don't allow the model any tool access at all…

I had to scroll too far to find this take. 100%.

This is like saying the CLAUDE.md or AGENTS.md is irrelevant because the LLM generated it.

Post reply on HN