Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SkillsBench: Benchmarking how well agent skills work across diverse tasks

arxiv.org

Post reply on HN
Print view View original

41–50 of 182 posts

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#41

Post by getoffit »

"Small models" will always outperform as they are deterministic (or closer to it).

This was realized in 2023 already: https://newsletter.semianalysis.com/p/google-we-have-no-moat...

"Less is best" is not a new realization. The concept exists across contexts. Music described as "overplayed". Prose described as verbose.

We just went through an era of compute that chanted "break down your monoliths". NPM ecosystem being lots of small little packages to compose together. Unix philosophy of small composable utilities is another example.

So models will improve as they are compressed, skeletonized down to opcodes, geometric models to render, including geometry for text as the bytecode patterns for such will provide the simplest model for recreating the most outputs. Compressing out useless semantics from the state of the machines operations and leaving the user to apply labels at the presentation layer.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#42

Post by paganel »

CharlieDigital wrote: post #32
wmeredith wrote: post #28

Earlier quoted context omitted.

> Agentic coding is the future, but people have not yet adapted. We went from punch cards to assembly to FORTRAN to C to JavaScript; each step adding more abstractions. I don't completely disagree (I've argued the same point myself). But one critical difference between the LLM layer and all of those others you listed, is that LLMs are non-deterministic and all those other layers are. I'm not sure how that changes the…

The LLM can be non-deterministic, but in the end, as long as we have compilers and integration tests, isn't it the same? You go from non-deterministic human interpretation of requirements and specs into a compiled, deterministic state machine. Now you have a non-deterministic coding agent doing the same and simply replacing the typing portion of that work. So long as you supply the agent well-curated set of guidance,…

The unspoken truth is that tests were never meant to cover all aspects of a piece of software running and doing its thing, that's where the "human mind(s)" that had actually built the system and brought it to life was supposed to come in and add the real layer of veracity. In other words, "if it walks like a duck and quacks like duck" was never enough, no matter how much duck-related testing was in place.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#43

Post by jerf »

siva7 wrote: post #40
j45 wrote: post #36

Earlier quoted context omitted.

Sincerely, perhaps you should publish on arxiv before a researcher reads it to run it and write a study. It's fairly common we notice these types of threads where one thing is being postulated and then there's comments upon comments of doer's showing what they have done.

somehow sad that some random dude on hn seems to have more brain than most scientists publishing on something about agents or prompting.

The AI world moves at a blistering pace. Academic publishing does not. In this particular case the "random dude on HN" is probably six to nine months ahead of the academic publication, not in the sense of being that much smarter but literally just being that much further progressed through time relative to the academic publication pipeline.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#44

Post by ljm »

btown wrote: post #30
dcre wrote: post #8

"Self-Generated Skills: No Skills provided, but the agent is prompted to generate relevant procedural knowledge before solving the task. This isolates the impact of LLMs’ latent domain knowledge" This is a useful result, but it is important to note that this is not necessarily what people have in mind when they think of "LLMs generating skills." Having the LLM write down a skill representing the lessons from the stru…

It's even worse than this: the "tasks" that are evaluated are limited to a single markdown file of instructions, plus an opaque verifier (page 13-14). No problems involving existing codebases, refactors, or anything of the like, where the key constraint is that the "problem definition" in the broadest sense doesn't fit in context. So when we look at the prompt they gave to have the agent generate its own skills: > Im…

I don't see how "create an abstraction before attempting to solve the problem" will ever work as a decent prompt when you are not even steering it towards specifics.

If you gave this exact prompt to a senior engineer I would expect them to throw it back and ask wtf you actually want.

LLMs are not mind readers.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#46

Post by lukev »

This clarifies an important point for me.

The derivative of a LLM agent's capabilities (on its own) is negative. It's not that they can't do useful work -- it means that (for now) they require some level of input or steering.

If that were to change -- if an agent could consistently get better at what it does without intervention -- that would represent a true paradigm shift. An accelerating curve, rather than one trending back towards linearity.

This represents a necessary inflection point for any sort of AI "takeoff" scenario.

So this study is actually kind of important, even though it's a null result. Because the contra view would be immensely significant.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#47

Post by firtoz »

smcleod wrote: post #12

There is almost no point in telling an agent to build a skill without augmenting it's knowledge on the thing it's writing about as you're just piping output to input without expanding the information in the system. If you get an agent to perform a bunch of research online, distil that down to information that the models tend not to get right or is newer than what is in their training data or simply better aligns with…

I find it useful for it to automatically update skills after trying them out in the wild. It can then improve the skills with real feedback. Seems to work well but I didn't do real research on it.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#48

Post by verdverm »

small_model wrote: post #27
verdverm wrote: post #23

Earlier quoted context omitted.

putting stuff in markdown files is not "learning", it's called taking notes, like we've done for 1000s of years

I guess when I was in class and took notes, then reviewed them later I wasn't "learning" anything.

That later "learning" part is updating weights in your brain

What Ai's get is a cheat sheet for the session

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#49

Post by Exoristos »

lukev wrote: post #46

This clarifies an important point for me. The derivative of a LLM agent's capabilities (on its own) is negative. It's not that they can't do useful work -- it means that (for now) they require some level of input or steering. If that were to change -- if an agent could consistently get better at what it does without intervention -- that would represent a true paradigm shift. An accelerating curve, rather than one tre…

Just to save us all some time and trouble, I'll point out that that's never really going to happen.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#50

Post by ericol »

dcre wrote: post #8

"Self-Generated Skills: No Skills provided, but the agent is prompted to generate relevant procedural knowledge before solving the task. This isolates the impact of LLMs’ latent domain knowledge" This is a useful result, but it is important to note that this is not necessarily what people have in mind when they think of "LLMs generating skills." Having the LLM write down a skill representing the lessons from the stru…

> Having the LLM write down a skill representing the lessons from the struggle you just had to get something done is more typical (I hope) and quite different from what they're referring to

Just as of last week I had Claude build me a skill when I ask it to help me troubleshoot issues, and it came out quite good.

It did had some issues (Claude tends to o er specify over anecdotal data) but it's a strong step in the right direction.

Also, "skills" are too broad in my opinion. I have one (that Claude wrote) with my personal data that I have available when I analyze my workouts.

I think there's ample room for self-generated skills when you use a rather long exchange on a domain you plan to revisit, _specially_ when it comes to telling Claude what not to do.

Post reply on HN