[flagged]
SkillsBench: Benchmarking how well agent skills work across diverse tasks
21–30 of 182 posts
Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks
#22Skills seem to be a crutch until we get continual learning. Imagine you've been running an instance for 6 months and it still remembers when you told it was running on your linux server over ssh and not on your Mac.
Search works well for this today, no need for continuous learning Not even sure how you envision continuous learning, but if you mean model updates, I'm not sure the economics work out
Actually claude has memory files now so it has some sort of learning, I think it will improve over time and they should survive a model update.
Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks
#23Earlier quoted context omitted.
Search works well for this today, no need for continuous learning Not even sure how you envision continuous learning, but if you mean model updates, I'm not sure the economics work out
Actually claude has memory files now so it has some sort of learning, I think it will improve over time and they should survive a model update.
putting stuff in markdown files is not "learning", it's called taking notes, like we've done for 1000s of years
Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks
#24"Self-Generated Skills: No Skills provided, but the agent is prompted to generate relevant procedural knowledge before solving the task. This isolates the impact of LLMs’ latent domain knowledge" This is a useful result, but it is important to note that this is not necessarily what people have in mind when they think of "LLMs generating skills." Having the LLM write down a skill representing the lessons from the stru…
Yeah I care about LLM's generating skills after attempting tasks and learning lessons from those attempts, not before attempting a task for the first time. This result seems a little silly and detached from the reality of how skills are "auto-generated" in the real world.
Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks
#25Earlier quoted context omitted.
Actually claude has memory files now so it has some sort of learning, I think it will improve over time and they should survive a model update.
putting stuff in markdown files is not "learning", it's called taking notes, like we've done for 1000s of years
Socrates made a similar complaint about the invention of writing, itself.
Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks
#26It seems intuitive that a naive self-generated Skill would be low-value, since the model already knows whatever it's telling itself. However, I've found them to be useful for capturing instructions on how to use other tools (e.g. hints on how to use command-line tools or APIs). I treat them like mini CLAUDE.mds that are specific only to certain workflows. When Claude isn't able to use a Skill well, I ask it to reflec…
A pattern I use a lot is after working with the LLM on a problem, directing it, providing additional context and information, ask it to summarize its learning into a skill. Then the next session that has a similar theme can start with that knowledge.
Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks
#27Earlier quoted context omitted.
Actually claude has memory files now so it has some sort of learning, I think it will improve over time and they should survive a model update.
putting stuff in markdown files is not "learning", it's called taking notes, like we've done for 1000s of years
I guess when I was in class and took notes, then reviewed them later I wasn't "learning" anything.
Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks
#28This has been my observation with self-generated docs as well. I have seen some devs pull out absolutely bad guidance by introspecting the code with the LLM to define "best practices" and docs because it introduces its own encoded biases in there. The devs are so lazy that they can't be bothered to simply type the bullet points that define "good". One example is that we had some extracted snippet for C#/.NET that was…
> Agentic coding is the future, but people have not yet adapted. We went from punch cards to assembly to FORTRAN to C to JavaScript; each step adding more abstractions.
I don't completely disagree (I've argued the same point myself). But one critical difference between the LLM layer and all of those others you listed, is that LLMs are non-deterministic and all those other layers are. I'm not sure how that changes the dynamic, but surely it does.
Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks
#29The general rule seems to be, the more layers you automate with LLMs, the worse each successive layer gets. Piping LLM output as input into new LLM calls, you're already starting to notice how things fall apart and get lost quickly. If you have the idea, more or less the implementation plan, let the LLM do the coding, you can end up with something maintainable and nice, it's basically up to you. Strip away one layer,…
People like to make the comparison between zip file compressions, where you can degrade something by continually compressing. Same with using jpeg or mp3. But I like to use the analogy of the game "Telephone" (also called "Chinese Whispers"). I think it also highlights how fraught natural language is and just how quickly it can degrade. I think a lot of people are insufficiently impressed with how good we are at comm…
ZIP files are lossless. If you compress, unzip, and recompress a ZIP file hundreds of times, it'll still be the exact same data as when you started.
Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks
#30"Self-Generated Skills: No Skills provided, but the agent is prompted to generate relevant procedural knowledge before solving the task. This isolates the impact of LLMs’ latent domain knowledge" This is a useful result, but it is important to note that this is not necessarily what people have in mind when they think of "LLMs generating skills." Having the LLM write down a skill representing the lessons from the stru…
It's even worse than this: the "tasks" that are evaluated are limited to a single markdown file of instructions, plus an opaque verifier (page 13-14). No problems involving existing codebases, refactors, or anything of the like, where the key constraint is that the "problem definition" in the broadest sense doesn't fit in context.
So when we look at the prompt they gave to have the agent generate its own skills:
> Important: Generate Skills First Before attempting to solve this task, please follow these steps: 1. Analyze the task requirements and identify what domain knowledge, APIs, or techniques are needed. 2. Write 1–5 modular skill documents that would help solve this task. Each skill should: focus on a specific tool, library, API, or technique; include installation/setup instructions if applicable; provide code examples and usage patterns; be reusable for similar tasks. 3. Save each skill as a markdown file in the environment/skills/ directory with a descriptive name. 4. Then solve the task using the skills you created as reference.
There's literally nothing it can do by way of "exploration" to populate and distill self-generated skills - not with a web search, not exploring an existing codebase for best practices and key files - only within its own hallucinations around the task description.
It also seems they're not even restarting the session after skills are generated, from that fourth bullet? So it's just regurgitating the context that was used to generate the skills.
So yeah, your empty-codebase vibe coding agent can't just "plan harder" and make itself better. But this is a misleading result for any other context, including the context where you ask for a second feature on that just-vibe-coded codebase with a fresh session.