How is this related to the bitter lesson?
The Bitter Lesson of LLM Extensions
31–40 of 81 posts
Re: The Bitter Lesson of LLM Extensions
#32Well, that's just great.
The academic community has been using the term "skill" for years, to refer to classes of tasks at which LLMs exhibit competence.
Now OpenAI has usurped the term to refer to these inference-guiding .md files.
I'm not looking forward to having to pick through a Google hit list for "LLM skills", figuring out which publications are about skills in the traditional sense and which are about the OpenAI feature. Semantic overload sucks.
How do we deal with this? Start using "competencies" (or similar) in academic papers? Or just resign ourselves to suffering the ambiguity?
Or maybe the OpenAI feature will fall flat and nobody will talk about it at all. That would frankly be the best outcome.
Re: The Bitter Lesson of LLM Extensions
#33Well, that's just great. The academic community has been using the term "skill" for years, to refer to classes of tasks at which LLMs exhibit competence. Now OpenAI has usurped the term to refer to these inference-guiding .md files. I'm not looking forward to having to pick through a Google hit list for "LLM skills", figuring out which publications are about skills in the traditional sense and which are about the Ope…
The way NNs and LLMs solve this problem is by processing context and activating middle layer nodes to disambiguate local ambiguities. Have you tried increasing your context window?
Re: The Bitter Lesson of LLM Extensions
#34Earlier quoted context omitted.
I still don't get what is special about the skills directory - since like forever I instructed Claud Code - "please read X and do Y" - how skills are different from that?
They're not. They are just a formalization of that pattern, with a very tiny extra feature where the model harness scans that folder on startup and loads some YAML metadata into the system prompt so it knows which ones to read later on.
So "skills" are a hack around the LLM not actually being very smart? Interesting.
Re: The Bitter Lesson of LLM Extensions
#35> Skills are the actualization of the dream that was set out by ChatGPT Plugins .. But I have a hypothesis that it might actually work now because the models are actually smart enough for it to work. and earlier Simon Willison argued[1] that Skills are even bigger deal than MCP. But I do not see as much hype for Skills as it was for MCP - it seems people are in the MCP "inertia" and having no time to shift to Skills.…
Skills are less exciting because they're effectively documentation that's selectively loaded.
They are a bigger deal in a sense because they remove the need for all the scaffolding MCPs require.
E.g. I needed Claude to work on transcripts from my Fathom account, so I just had it write a CLI script to download them, and then I had it write a SKILL.md, and didn't have to care about wrapping it up into an MCP.
At a client, I needed a way to test their APIs, so I just told Claude Code to pull out the client code from one of their projects and turn it into a CLI, and then write a SKILL.md. And again, no need to care about wrapping it up into an MCP.
But this seems a lot less remarkable, and there's a lot less room to build big complicated projects and tooling around it, and so, sure, people will talk about it less.
Re: The Bitter Lesson of LLM Extensions
#36The most useful LLM "extension" isn't even mentioned in this article, and that is shell use.
An LLM with a shell integration can do anything you need it to.
Re: The Bitter Lesson of LLM Extensions
#37Earlier quoted context omitted.
They're not. They are just a formalization of that pattern, with a very tiny extra feature where the model harness scans that folder on startup and loads some YAML metadata into the system prompt so it knows which ones to read later on.
So "skills" are a hack around the LLM not actually being very smart? Interesting.
It's more that they are embracing that the LLM is smart enough that you don't need to build-in this functionality beyond that very minimal part.
A fun thing: Claude Code will sometimes fail to find the skill the "proper" way, and will then in fact sometimes look for the SKILL.md file with tools, and read the file with tools, showing that it's perfectly capable of doing all the steps.
You could probably "fake" skills pretty well with instructions in CLAUDE.md to use a suitable command to extract the preamble of files in a given directory, and tell it to use that to decide when to read the rest.
It's the fact that it's such a thin layer that is exciting - it means we need increasingly less special logic other than relying on just basic instructions to the model itself.
Re: The Bitter Lesson of LLM Extensions
#38Earlier quoted context omitted.
The difference is that the code in the directory (and the markdown) are hardcoded and known to work beforehand.
But we are still reliant on the LLM correctly interpreting the choice to pick the right skill. So "known to work" should be understood in the very limited context of "this sub-function will do what it was designed to do reliably" rather than "if the user asks to use this sub-function it will do was it was designed to do reliably". Skills feel like a non-feature to me. It feels more valuable to connect a user to the a…
This is no different to an MCP, where you rely on the model to use the metadata provided to pick the right tool, and understand how to use it.
Like with MCP, you can provide a deterministic, known-good piece of code to carry out the operation once the LLM decides to use it.
But a skill can evolve from pure Markdown via inlining some shell commands, up to a large application. And if you let it, with Skills the LLM can also inspect the tool, and modify it if it will help you.
All the Skills I use now have evolved bit by bit as I've run into new use-cases and told Claude Code to update the script the skills references or the SKILL.md itself. I can evolve the tooling while I'm using it.
Re: The Bitter Lesson of LLM Extensions
#39The most useful LLM "extension" isn't even mentioned in this article, and that is shell use. An LLM with a shell integration can do anything you need it to.
A man with a spoon can dig a swimming pool but Id prefer a backhoe
Re: The Bitter Lesson of LLM Extensions
#40> Skills are the actualization of the dream that was set out by ChatGPT Plugins .. But I have a hypothesis that it might actually work now because the models are actually smart enough for it to work. and earlier Simon Willison argued[1] that Skills are even bigger deal than MCP. But I do not see as much hype for Skills as it was for MCP - it seems people are in the MCP "inertia" and having no time to shift to Skills.…
Skills are like the "end-user" version of MCP at best, where MCP is for people building systems. Any other point of view raises a lot of questions. Aren't skills really just a collection of tagged MCP prompts, config resources, and tools, except with more lock-in since only Claude can use it? About that "agent virtual environment" that runs the scripts.. how is it customized, and.. can it just be a container? Aren't…
There's no lock-in there.
Tell your agent of choice to read the preamble of all the documents in the skills directory, and tell it that when it has a task that matches one of the preambles, it should read the rest of the relevant file for full instructions.
There are far fewer dependencies for skills than for MCP. Even a model that knows nothing about tool use beyond how to run a shell command, and has no support for anything else can figure out skills.
I don't know what you mean regarding explicitly referencing other skills - Claude at least is smart enough that if you reference a skill that isn't even properly registered, it will often start using grep and find to hunt for it to figure out what you meant. I've seen this happen regularly while developing a plugin and having errors in my setup.