Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SkillsBench: Benchmarking how well agent skills work across diverse tasks

arxiv.org

Post reply on HN
Print view View original

11–20 of 182 posts

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#11

Post by godelski »

embedding-shape wrote: post #2

The general rule seems to be, the more layers you automate with LLMs, the worse each successive layer gets. Piping LLM output as input into new LLM calls, you're already starting to notice how things fall apart and get lost quickly. If you have the idea, more or less the implementation plan, let the LLM do the coding, you can end up with something maintainable and nice, it's basically up to you. Strip away one layer,…

People like to make the comparison between zip file compressions, where you can degrade something by continually compressing. Same with using jpeg or mp3. But I like to use the analogy of the game "Telephone" (also called "Chinese Whispers"). I think it also highlights how fraught natural language is and just how quickly it can degrade. I think a lot of people are insufficiently impressed with how good we are at communicating at all.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#12

Post by smcleod »

There is almost no point in telling an agent to build a skill without augmenting it's knowledge on the thing it's writing about as you're just piping output to input without expanding the information in the system. If you get an agent to perform a bunch of research online, distil that down to information that the models tend not to get right or is newer than what is in their training data or simply better aligns with your desired workflow than what they generate out of the box - that's going to create a far more useful skill. I use a skill that gets activated when creating a skill to help guide this approach: https://github.com/sammcj/agentic-coding/blob/main/Skills/sk...

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#13

Post by evmaki »

turnsout wrote: post #4

It seems intuitive that a naive self-generated Skill would be low-value, since the model already knows whatever it's telling itself. However, I've found them to be useful for capturing instructions on how to use other tools (e.g. hints on how to use command-line tools or APIs). I treat them like mini CLAUDE.mds that are specific only to certain workflows. When Claude isn't able to use a Skill well, I ask it to reflec…

> I ask it to reflect on why, and update the Skill to clarify, adding or removing detail as necessary.

We are probably undervaluing the human part of the feedback loop in this discussion. Claude is able to solve the problem given the appropriate human feedback — many then jump to the conclusion that well, if Claude is capable of doing it under some circumstances, we just need to figure out how to remove the human part so that Claude can eventually figure it out itself.

Humans are still serving a very crucial role in disambiguation, and in centering the most salient information. We do this based on our situational context, which comes from hands-on knowledge of the problem space. I'm hesitant to assume that because Claude CAN bootstrap skills (which is damn impressive!), it would somehow eventually do so entirely on its own, devoid of any situational context beyond a natural language spec.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#14

Post by sweetjuly »

godelski wrote: post #11
embedding-shape wrote: post #2

The general rule seems to be, the more layers you automate with LLMs, the worse each successive layer gets. Piping LLM output as input into new LLM calls, you're already starting to notice how things fall apart and get lost quickly. If you have the idea, more or less the implementation plan, let the LLM do the coding, you can end up with something maintainable and nice, it's basically up to you. Strip away one layer,…

People like to make the comparison between zip file compressions, where you can degrade something by continually compressing. Same with using jpeg or mp3. But I like to use the analogy of the game "Telephone" (also called "Chinese Whispers"). I think it also highlights how fraught natural language is and just how quickly it can degrade. I think a lot of people are insufficiently impressed with how good we are at comm…

I suggest you find a new DEFLATE library if you're losing data when you compress things with it :)

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#15

Post by verdverm »

Anecdotal middle ground, I have used LLM automation to generate AGENTS.md files at scale across a repo

1. You MUST review and correct them

2. Embrace minimalism, they are spark notes and an index, not comprehensive

3. Force them into context

I imagine similar concepts hold for skills

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#16

Post by verdverm »

ed_elliott_asc wrote: post #10
rrvsh wrote: post #7

Despite skills being just a new form of memory and context engineering for an agent, I think the framework is still great for agents to self-develop, given a good prompt to regularly review their own sessions and pick learning points to save as skills. In fact, I think the "craft" of prompt engineering has been lost somewhat - I still enjoy puzzling out and iterating over the best possible starting prompt for a conve…

Develop an ai skill to read articles and come up with a HN post for you :)

ai;dr (didn't read)

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#18

Post by embedding-shape »

quotemstr wrote: post #6
embedding-shape wrote: post #5

Earlier quoted context omitted.

But these are inherently subjective things, what the "right idea" is, or the "right implementation" is all up in our head that we can try to verbalize, but I don't think you can come up with an objective score for it, ask 100 programmers you'll get 100 different answers what "clean design" is.

And that's why my whole schtick when it comes to agent design is that agents need to learn online, continuously, and in adapter space via some PEFT mechanism (I like soft prompts and prefix tuning), because it's really hard to ascend gradients in discrete domains like tokens.

> The model knows damn well when it's written ugly code. You can just ask it.

That's not been my experience at all, what model and prompt would you use for that? Every single one I've tried is oblivious to if a design makes sense or not unless explicitly prompted for it with constraints, future ideas and so on.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#19

Post by verdverm »

small_model wrote: post #17

Skills seem to be a crutch until we get continual learning. Imagine you've been running an instance for 6 months and it still remembers when you told it was running on your linux server over ssh and not on your Mac.

Search works well for this today, no need for continuous learning

Not even sure how you envision continuous learning, but if you mean model updates, I'm not sure the economics work out

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#20

Post by CharlieDigital »

This has been my observation with self-generated docs as well.

I have seen some devs pull out absolutely bad guidance by introspecting the code with the LLM to define "best practices" and docs because it introduces its own encoded biases in there. The devs are so lazy that they can't be bothered to simply type the bullet points that define "good".

One example is that we had some extracted snippet for C#/.NET that was sprinkling in `ConfigureAwait(false)` which should not be in application code and generally not needed for ASP.NET. But the coding agent saw some code that looked like "library" code and decided to apply it and then someone ran the LLM against that and pulled out "best practices" and placed them into the repo and started to pollute the rest of the context.

I caught this when I found the code in a PR and then found the source and zeroed it out. We've also had to untangle some egregious use of `Task.Run` (again, not best practice in C# and you really want to know what you're doing with it).

At the end of it, we are building a new system that is meant to compose and serve curated, best practice guidance to coding agents to get better consistency and quality. The usage of self-generated skills and knowledge seems like those experiments where people feed in an image and ask the LLM to give back the image without changing it. After n cycles, it is invariably deeply mutated from the original.

Agentic coding is the future, but people have not yet adapted. We went from punch cards to assembly to FORTRAN to C to JavaScript; each step adding more abstractions. The next abstraction is Markdown and I think that teams that invest their time in writing and curating markdown will create better guardrails within which agents can operate without sacrificing quality, security, performance, maintainability, and other non-functional aspects of software system.

Post reply on HN