Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Bitter Lesson of LLM Extensions

sawyerhood.com

Post reply on HN
Print view View original

21–30 of 81 posts

Re: The Bitter Lesson of LLM Extensions

#21

Post by munk-a »

mkagenius wrote: post #12
zby wrote: post #6

Earlier quoted context omitted.

I still don't get what is special about the skills directory - since like forever I instructed Claud Code - "please read X and do Y" - how skills are different from that?

The difference is that the code in the directory (and the markdown) are hardcoded and known to work beforehand.

But we are still reliant on the LLM correctly interpreting the choice to pick the right skill. So "known to work" should be understood in the very limited context of "this sub-function will do what it was designed to do reliably" rather than "if the user asks to use this sub-function it will do was it was designed to do reliably".

Skills feel like a non-feature to me. It feels more valuable to connect a user to the actual tool and let them familiarize themselves with it (and not need the LLM to find it in the future) rather than having the tool embedded in the LLM platform. I will carve out a very big exception of accessibility here - I love my home device being an egg timer - it's a wonderful egg timer (when it doesn't randomly play music) and I could buy an egg timer but having a hands-free egg timer is actually quite valuable to me while cooking. So I believe there is real value in making these features accessible through the LLM over media that the feature would normally be difficult to use in.

Re: The Bitter Lesson of LLM Extensions

#22

Post by munk-a »

mkagenius wrote: post #15
fzysingularity wrote: post #9

Earlier quoted context omitted.

I definitely see the value and versatility of Claude Skills (over what MCP is today), but I find the sandboxed execution to be painfully inefficient. Even if we expect the LLMs to fully resolve the task, it'll heavily rely on I/O and print statements sprinkled across the execution trace to get the job done.

> but I find the sandboxed execution to be painfully inefficient sandbox is not mandatory here. You can execute the skills on your host machine too (with some fidgeting) but it's a good practice and probably for the better to get in to the habit of executing code in an isolated environment for security purposes.

The better practice is, if it isn't a one-off, being introduced to the tool (perhaps by an LLM) and then just running the tool yourself with structured inputs when it is appropriate. I think the 2015 era novice coding habit of copying a blob of twenty shell scripts off of stack overflow and blindly running them in your terminal (while also not good for obvious reasons) was better than that essentially happening but you not being able to watch and potentially learn what those commands were.

Re: The Bitter Lesson of LLM Extensions

#23

Post by vessenes »

robot-wrangler wrote: post #13
vessenes wrote: post #3

> "If I could short MCP, I would" I mean, MCP is hard to work with. But there's a very large set of things that we want a hardened interface to out there - if not MCP, it will be something very like it. In particular, MCP was probably overly complicated at the design phase to deal with the realities of streaming text / tokens back and forth live. That is, it chose not to abstract these realities in exchange for some…

I always see the hard/complex criticism but find it confusing.. what is the perceived difficulty with MCP at the implementation level? (I do understand the criticism about exhausting tokens with tool-descriptions and stuff, but that's a different challenge) Doesn't seem like implementation could be more simple. Just JSON-RPC and API stuff. For example the MCP hello-world with python and FastMCP is practically 1-to-1…

There is a LOT under the surface. custom routes, bidirectional streaming choices (it started as a "local first" protocol). Implementing an endpoint from scratch is not easy, and the spec documentation moves very quickly, and generally doesn't have simple-to-digest updates for implementation.

I haven't looked in a few months, so my information might be a bit out of date, but at the time - if you wanted to use a python server from the modelcontextprotocol GitHub, fine. If you wanted to, say, build a proxy server in rust or golang, you were looking at a set of half-implemented server implementations targeting two-versions-old MCP specs while clients like claude obscure even which endpoints they use for discovery.

It's an immature spec, moderately complicated, and moving really quickly with only a few major 'subscribers' to the server side; I found it challenging to work with.

Re: The Bitter Lesson of LLM Extensions

#24

Post by fzysingularity »

munk-a wrote: post #22
mkagenius wrote: post #15

Earlier quoted context omitted.

> but I find the sandboxed execution to be painfully inefficient sandbox is not mandatory here. You can execute the skills on your host machine too (with some fidgeting) but it's a good practice and probably for the better to get in to the habit of executing code in an isolated environment for security purposes.

The better practice is, if it isn't a one-off, being introduced to the tool (perhaps by an LLM) and then just running the tool yourself with structured inputs when it is appropriate. I think the 2015 era novice coding habit of copying a blob of twenty shell scripts off of stack overflow and blindly running them in your terminal (while also not good for obvious reasons) was better than that essentially happening but y…

I do think that if the agents can successfully resolve these tasks in a code execution environment, it can likely come up with better parametrized solutions with structured I/O - assuming these are workflows we want to run over and over again.

Re: The Bitter Lesson of LLM Extensions

#25

Post by sawyerjhood »

mkagenius wrote: post #4

> Skills are the actualization of the dream that was set out by ChatGPT Plugins .. But I have a hypothesis that it might actually work now because the models are actually smart enough for it to work. and earlier Simon Willison argued[1] that Skills are even bigger deal than MCP. But I do not see as much hype for Skills as it was for MCP - it seems people are in the MCP "inertia" and having no time to shift to Skills.…

I agree with you. I don't see people hyping them and I think a big part of this is that we have sort of hit an LLM fatigue point right now. Also Skills require that your agent can execute arbitrary code which is a bigger buy-in cost if your app doesn't have this already.

Re: The Bitter Lesson of LLM Extensions

#27

Post by robot-wrangler »

vessenes wrote: post #23
robot-wrangler wrote: post #13

Earlier quoted context omitted.

I always see the hard/complex criticism but find it confusing.. what is the perceived difficulty with MCP at the implementation level? (I do understand the criticism about exhausting tokens with tool-descriptions and stuff, but that's a different challenge) Doesn't seem like implementation could be more simple. Just JSON-RPC and API stuff. For example the MCP hello-world with python and FastMCP is practically 1-to-1…

There is a LOT under the surface. custom routes, bidirectional streaming choices (it started as a "local first" protocol). Implementing an endpoint from scratch is not easy, and the spec documentation moves very quickly, and generally doesn't have simple-to-digest updates for implementation. I haven't looked in a few months, so my information might be a bit out of date, but at the time - if you wanted to use a python…

Well if your language of choice didn't have any good library support for HTTP, the web version of hello world would be hard too, but it would not say much about the protocol.

Even with these constraints the core MCP design is actually pretty good. First, use stdio transport, and now your language only needs to speak JSON [1]. Then, forget about building proxies and routers and web stuff, and offload that to mcpjungle [2] or similar to front your stdio work.

If that still doesn't work, I think I would probably wrap the foreign language with subprocs and retreat towards python's FastMCP (or whatever the well-supported and fast-moving stuff is in another language). Ugly but practical if you really must use a language with no good MCP support. If really none of that works I guess one is on the hook to support a changing MCP spec with a custom implementation in that language.. but isn't there maybe an argument now that MCP is complex because someone insisted on it being complex?

[1]: https://modelcontextprotocol.io/specification/2025-06-18/bas... [2]: https://github.com/mcpjungle/MCPJungle

Re: The Bitter Lesson of LLM Extensions

#30

Post by mkagenius »

munk-a wrote: post #21
mkagenius wrote: post #12

Earlier quoted context omitted.

The difference is that the code in the directory (and the markdown) are hardcoded and known to work beforehand.

But we are still reliant on the LLM correctly interpreting the choice to pick the right skill. So "known to work" should be understood in the very limited context of "this sub-function will do what it was designed to do reliably" rather than "if the user asks to use this sub-function it will do was it was designed to do reliably". Skills feel like a non-feature to me. It feels more valuable to connect a user to the a…

Choice to pick right tool -- there is a benchmark which tracks the accuracy of this.

"Known to work" -- if it has a hardcoded code, it will work 100% of the time - that's the point of Skills. If it's just markdown then yes, some sort of probability will be there and it will keep on improving.

Post reply on HN