Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SkillsBench: Benchmarking how well agent skills work across diverse tasks

arxiv.org

Post reply on HN
Print view View original

171–180 of 182 posts

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#171

Post by xdotli »

dalemhurley wrote: post #125
isahers wrote: post #24

Earlier quoted context omitted.

Yeah I care about LLM's generating skills after attempting tasks and learning lessons from those attempts, not before attempting a task for the first time. This result seems a little silly and detached from the reality of how skills are "auto-generated" in the real world.

That is my approach. I don’t think the papers author has actually used skills.

Did you check our repos and sites? the repo is skills native. Also please don't be misled by the original title, we have this configuration to eliminate the impact of internal knowledge of LLMs. It's in the paper.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#172

Post by xdotli »

mustaphah wrote: post #169
xdotli wrote: post #167

Earlier quoted context omitted.

Thanks @dang for moderating! This is indeed not our original findings and this is a sub conclusion for an ablation we did to remove the confound of LLMs internal domain knowledge. Thanks for submitting for us @mustaphah here's a little bit more details on how we approach this: > I would frame the 'post-trajectory generated skills' as feedback-generated skills, so is Letta: https://www.letta.com/blog/skill-learning .…

Yeah, I got your point when I read the paper. You're essentially controlling for "latent domain knowledge." I might have been a bit blunt with the title - sorry about that, but I still think it was a good title. From what I've observed, a lot of Skills on GitHub are just AI-generated without any feedback or deliberative refinement. Many thought those would still be valuable, but you've shown evidence otherwise.

no worries it's totally fine! there is indeed work needs to be done on the feedbacks generated skills. Thanks for helping us submitting on HackerNews. And for > a lot of Skills on GitHub are just AI-generated without any feedback or deliberative refinement. Many thought those would still be valuable, but you've shown evidence otherwise. we do find most skills on the internet to be useless, and thanks to the generosity of https://skillsmp.com/ author, we were able to get all the meta data of the 99k skills indexed on his website. We did a lot of filtering and deduping and we discovered ~40k+ skills were relevant at the time we did the study.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#173

Post by jwpapi »

8note wrote: post #117
rapind wrote: post #103

Earlier quoted context omitted.

CLAUDE.md in my projects only has coding / architecture guidelines. Here's what not to do. Here's what you should do. Here are my preferences. Here's where the important things are. Even though my CLAUDE.md is small though, often my rules are ignored. Not always though, so it's still at least somewhat useful!

im trying out some other cc features, and om thinking maybe hooks can do something with this. have a hook on switching out of plan, and maybe on edits, that passes the change to haiku with the claude.md to see if it matches or not

man that’s what I’m trying to build the whole time, but I keep getting json parsing errors. I’ve debugged a lot, but it seems their haiku is not consistent with the actual output. I want a hook that tells them at the end make sure you’ve built and run the relevant tests. Let me know if you need anything else.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#174

Post by btown »

ljm wrote: post #160
btown wrote: post #154

Earlier quoted context omitted.

If it were in the context of parachuting into a codebase, I’d make these skills an important familiarization exercise: how are tests made, what are patterns I see frequently, what are the most important user flows. By forcing myself to distill that first, I’d be better at writing code that is in keeping with the codebase’s style and overarching/subtle goals. But this makes zero sense in a green-field task.

There's overlap in that with brownfield or legacy code you are strongly opinionated on the status quo, and on the greenfield you are strongly opinionated with fewer constraints. You have to work with conviction though. It's when you offload everything to the LLM that things start to drift from expectations, because you kept the expectations in your head and away from the prompt.

Do skills extracted from existing codebases cause better or worse code in that they bias the LLM towards existing bad practices? Or, can they assist in acknowledging these practices, and bias it towards actively ensuring they're fixed in new code? How dependent is this on the prompt used for the skill extraction? Are the skills an improvement over just asking to do this extraction at the start of the task?

Now this dynamic would be a good topic to research!

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#175

Post by ethmarks »

godelski wrote: post #111
sweetjuly wrote: post #14

Earlier quoted context omitted.

I suggest you find a new DEFLATE library if you're losing data when you compress things with it :)

You do realize there is both lossy and lossless compression, right? Or did you hyperfixate on the colloquial usage of zip

> You do realize there is both lossy and lossless compression, right?

This is simply wrong. There is no lossy compression for ZIP.

DEFLATE, by far the most common ZIP compression method, uses LZ77 and Huffman Coding, both of which are lossless. There are other methods compatible with ZIP containers as specified by PKWARE (e.g. BZIP2, LZMA, Zstandard, PPMd, etc), but all of them are lossless. According to both the official ZIP specification and every ZIP implementation on Earth, you cannot have a lossy ZIP unless it is corrupted.

There do exist lossy data formats (e.g. JPEG), but if you put those in a ZIP file it'll still encode and decode it losslessly.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#176

Post by defrost »

ethmarks wrote: post #175
godelski wrote: post #111

Earlier quoted context omitted.

You do realize there is both lossy and lossless compression, right? Or did you hyperfixate on the colloquial usage of zip

> You do realize there is both lossy and lossless compression, right? This is simply wrong. There is no lossy compression for ZIP. DEFLATE, by far the most common ZIP compression method, uses LZ77 and Huffman Coding, both of which are lossless. There are other methods compatible with ZIP containers as specified by PKWARE (e.g. BZIP2, LZMA, Zstandard, PPMd, etc), but all of them are lossless. According to both the off…

> There is no lossy compression for ZIP.

Did you hyperfixate on the colloquial usage of zip?

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#177

Post by ethmarks »

defrost wrote: post #176
ethmarks wrote: post #175

Earlier quoted context omitted.

> You do realize there is both lossy and lossless compression, right? This is simply wrong. There is no lossy compression for ZIP. DEFLATE, by far the most common ZIP compression method, uses LZ77 and Huffman Coding, both of which are lossless. There are other methods compatible with ZIP containers as specified by PKWARE (e.g. BZIP2, LZMA, Zstandard, PPMd, etc), but all of them are lossless. According to both the off…

> There is no lossy compression for ZIP. Did you hyperfixate on the colloquial usage of zip?

> Did you hyperfixate on the colloquial usage of zip?

No? I am not hyperfixating on the colloquial usage of ZIP. The colloquial usage of "zip" would be "any compression container", which is not what I'm talking about. I'm talking about the technical definition of ZIP: the lossless container format specified by PKWARE. I thought that would be obvious by my reference to the PKWARE specification.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#178

Post by defrost »

ethmarks wrote: post #177
defrost wrote: post #176

Earlier quoted context omitted.

> There is no lossy compression for ZIP. Did you hyperfixate on the colloquial usage of zip?

> Did you hyperfixate on the colloquial usage of zip? No? I am not hyperfixating on the colloquial usage of ZIP. The colloquial usage of "zip" would be "any compression container", which is not what I'm talking about. I'm talking about the technical definition of ZIP: the lossless container format specified by PKWARE. I thought that would be obvious by my reference to the PKWARE specification.

@godelski clearly indicated 8 days ago they were not referring to to formal ZIP files.

Your misunderstanding might be due to not being exposed to long standing casual usage of "zipping" something up to include many archiving formats, a number of which do include lossy compression methods.

Congrats on being a 15 year old developer, though .. some of the folk kicking about on HN wrote the books(?) you may have read.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#179

Post by ethmarks »

defrost wrote: post #178
ethmarks wrote: post #177

Earlier quoted context omitted.

> Did you hyperfixate on the colloquial usage of zip? No? I am not hyperfixating on the colloquial usage of ZIP. The colloquial usage of "zip" would be "any compression container", which is not what I'm talking about. I'm talking about the technical definition of ZIP: the lossless container format specified by PKWARE. I thought that would be obvious by my reference to the PKWARE specification.

@godelski clearly indicated 8 days ago they were not referring to to formal ZIP files. Your misunderstanding might be due to not being exposed to long standing casual usage of "zipping" something up to include many archiving formats, a number of which do include lossy compression methods. Congrats on being a 15 year old developer, though .. some of the folk kicking about on HN wrote the books(?) you may have read.

Putting aside both my age and the latency of my initial response, all that I was trying to do was correct godelski's erroneous attempted correction of sweetjuly's lighthearted joke. The "colloquial usage of zip" that godelski was chiding sweetjuly for "hyperfixating" on is exactly the usage that godelski was using in their original reply. I did not misunderstand godelski's intended use: they were using "zip" to refer to "any compression format whatsoever, be it lossy or lossless". This is a technically incorrect usage, which would be fine if not for the snarky chiding. I think that misusing a term and then accusing people of "hyperfixating" when they lightheartedly correct you isn't a particularly nice thing to do.

Re: SkillsBench: Benchmarking how well agent skills work across diverse tasks

#180

Post by defrost »

ethmarks wrote: post #179
defrost wrote: post #178

Earlier quoted context omitted.

@godelski clearly indicated 8 days ago they were not referring to to formal ZIP files. Your misunderstanding might be due to not being exposed to long standing casual usage of "zipping" something up to include many archiving formats, a number of which do include lossy compression methods. Congrats on being a 15 year old developer, though .. some of the folk kicking about on HN wrote the books(?) you may have read.

Putting aside both my age and the latency of my initial response, all that I was trying to do was correct godelski's erroneous attempted correction of sweetjuly's lighthearted joke. The "colloquial usage of zip" that godelski was chiding sweetjuly for "hyperfixating" on is exactly the usage that godelski was using in their original reply. I did not misunderstand godelski's intended use: they were using "zip" to refer…

\1 godelski colloquially used the term zip (Not referring to formal ZIP files)

\2 sweetjuly made reference to ZIP files not having lossy methods

\3 godelski indicated they had use zip colloquially, not with the intention of referencing the precise ZIP specification.

\4 ethmarks stepped in and doubled down on the precise ZIP specification, despite that not being what godelski initially referred to.

Here's the thing, people make casual remarks, they use imprecise non technical colloquillisms.

How would you describe the actions in \4 ? Do you believe it served a useful purpose to double down and restate something that was very likely well known to both parties over a week ago?

Post reply on HN