Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SkillsBench: Benchmarking how well agent skills work across diverse tasks

arxiv.org

Post reply on HN
Print view View original

101–110 of 182 posts

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#101

Post by jibal »

small_model wrote: post #27
verdverm wrote: post #23

Earlier quoted context omitted.

putting stuff in markdown files is not "learning", it's called taking notes, like we've done for 1000s of years

I guess when I was in class and took notes, then reviewed them later I wasn't "learning" anything.

Reading notes is only necessary because of how lossy human memory is. Reading notes doesn't give you new information, it just reinforces memory paths ... which will fade and you'll have to read the notes again later unless you frequently apply the knowledge, which again reinforces those paths (but lossily, so the bits of information not repeatedly used will fade, and you will again have to read the notes if you need those bits ... or just to re-mind yourself what they were).

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#102

Post by gwern »

zahlman wrote: post #63
rahimnathwani wrote: post #31

This is unsurprising and irrelevant. When you create a skill for a particular model, you don't typically ask the model to create the skill based solely on its own latent knowledge. Otherwise, you'd expect the effect to be similar to telling the model 'make a plan before acting, make not mistakes'. But that's what the paper's authors did! When they say 'self-generated' they don't allow the model any tool access at all…

> Otherwise, you'd expect the effect to be similar to telling the model 'make a plan before acting, make not mistakes'. Have there not been previous iterations of these tools where such techniques were actually effective?

But that's a reason you should expect it to stop working soon, just like all the older tricks like "my grandmother will die". If you have a universal 'blind' prompt which can increase performance a little bit... the AI labs can just toss that into the training loop to teach the model to do it automatically, whatever 'it' was, like 'trying harder' or 'writing down a useful idea'. And then the prompt stops working because the next generations do it by default.

(This also suggests that you should expect them to generally be bad at judging novel self-generated prompts/skills - if they could judge those, they would already be using them! There is a generator-verifier gap, but it is already exploited heavily during post-training and not much low-hanging fruit left there.)

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#103

Post by rapind »

jwpapi wrote: post #89
btown wrote: post #30

Earlier quoted context omitted.

It's even worse than this: the "tasks" that are evaluated are limited to a single markdown file of instructions, plus an opaque verifier (page 13-14). No problems involving existing codebases, refactors, or anything of the like, where the key constraint is that the "problem definition" in the broadest sense doesn't fit in context. So when we look at the prompt they gave to have the agent generate its own skills: > Im…

Thats actually super interesting and why I really don’t like the whole .md folder structures or even any CLAUDE.md. It just seems most of the time you really just want to give it what it needs for best results. The headline is really bullshit, yes, I like the testing tho.

CLAUDE.md in my projects only has coding / architecture guidelines. Here's what not to do. Here's what you should do. Here are my preferences. Here's where the important things are.

Even though my CLAUDE.md is small though, often my rules are ignored. Not always though, so it's still at least somewhat useful!

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#104

Post by embedding-shape »

CuriouslyC wrote: post #72
embedding-shape wrote: post #18

Earlier quoted context omitted.

> The model knows damn well when it's written ugly code. You can just ask it. That's not been my experience at all, what model and prompt would you use for that? Every single one I've tried is oblivious to if a design makes sense or not unless explicitly prompted for it with constraints, future ideas and so on.

The problem is that the model doesn't know what you mean by "bad code" a priori. If you list specific issues you care about (e.g. separation of concerns, don't repeat yourself, single responsibility, prefer pure functions, etc) it's pretty good at picking them out. Humans have this problem as well, we're just more opinionated.

Yes, that's exactly what I mentioned earlier, if you describe the implementation, you can get something you can work with long-term. But if you just describe an idea, and let the LLM do both the design of the implementation and the implementation itself, eventually it seems to fall over itself and changes takes longer and longer time.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#105

Post by jibal »

godelski wrote: post #11
embedding-shape wrote: post #2

The general rule seems to be, the more layers you automate with LLMs, the worse each successive layer gets. Piping LLM output as input into new LLM calls, you're already starting to notice how things fall apart and get lost quickly. If you have the idea, more or less the implementation plan, let the LLM do the coding, you can end up with something maintainable and nice, it's basically up to you. Strip away one layer,…

People like to make the comparison between zip file compressions, where you can degrade something by continually compressing. Same with using jpeg or mp3. But I like to use the analogy of the game "Telephone" (also called "Chinese Whispers"). I think it also highlights how fraught natural language is and just how quickly it can degrade. I think a lot of people are insufficiently impressed with how good we are at comm…

> People like to make the comparison between zip file compressions, where you can degrade something by continually compressing.

What people have this misunderstanding?

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#106

Post by nubg »

dcre wrote: post #8

"Self-Generated Skills: No Skills provided, but the agent is prompted to generate relevant procedural knowledge before solving the task. This isolates the impact of LLMs’ latent domain knowledge" This is a useful result, but it is important to note that this is not necessarily what people have in mind when they think of "LLMs generating skills." Having the LLM write down a skill representing the lessons from the stru…

> I'm sure news outlets and popular social media accounts will use appropriate caution in reporting this, and nobody will misunderstand it.

:D

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#107

Post by pixl97 »

gruez wrote: post #95
zozbot234 wrote: post #53

Earlier quoted context omitted.

AI companies don't want you to waste tokens, they benefit when you use them efficiently because they can serve more users on the infra that's the main bottleneck for them. It's Jevons' paradox in action.

>AI companies don't want you to waste tokens, they benefit when you use them efficiently because they can serve more users on the infra that's the main bottleneck for them. No, the actual incentive is that people will eventually benchmark their models on bang-per-buck basis and models that chew through tokens are not going to be competitive. It's the same reason why the "Intel/AMD are intentionally sandbagging their…

Well, it only works when one competitor is far enough ahead they can play games like that.

At least currently in AI there is no moat so we wouldn't expect that to be occurring

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#109

Post by awwaiid »

zozbot234 wrote: post #37
dcre wrote: post #8

"Self-Generated Skills: No Skills provided, but the agent is prompted to generate relevant procedural knowledge before solving the task. This isolates the impact of LLMs’ latent domain knowledge" This is a useful result, but it is important to note that this is not necessarily what people have in mind when they think of "LLMs generating skills." Having the LLM write down a skill representing the lessons from the stru…

The point of so-called 'skills' is to be short how-to reminders that the agent can pull into its context and then act upon. If the knowledge is already in the model, it will most likely be surfaced in reasoning phase anyway, so there's little benefit to writing it up as a skill, unless perhaps it's extremely relevant and hard to surface, and you want the model to skip that part of the reasoning.

I've been building a skill to help run manual tests on an app. So I go through and interactively steer toward a useful validation of a particular PR, navigating specifics of the app and what I care about and what I don't. Then in the end I have it build a skill that would have skipped backtracking and retries and the steering I did.

Then I do it again from scratch; this time it takes less steering. I have it update the skill further.

I've been doing this on a few different tests and building a skill which is taking less and steering to do app-specific and team-specific manual testing faster and faster. The first times through it took longer than manually testing the feature. While I've only started doing this recently, it is now taking less time than I would take, and posting screenshots of the results and testing steps in the PR for dev review. Ongoing exploration!

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#110

Post by daxfohl »

I'm kinda surprised by this. Yeah it's just regurgitating something it already "knows", but I'd still expect that having the skill materialized there in the context would give it something concrete to reference, less likelihood of getting lost or hallucinating, and probably need less incremental context to do the job.

I mean, basically it's doing the same thing as reasoning IIUC, except up-front rather than inline and ad-hoc, so I'd almost expect it to work even better than reasoning alone.

Post reply on HN