Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Bitter Lesson of LLM Extensions

sawyerhood.com

Post reply on HN
Print view View original

41–50 of 81 posts

Re: The Bitter Lesson of LLM Extensions

#41

Post by vidarh »

CuriouslyC wrote: post #7
mkagenius wrote: post #4

> Skills are the actualization of the dream that was set out by ChatGPT Plugins .. But I have a hypothesis that it might actually work now because the models are actually smart enough for it to work. and earlier Simon Willison argued[1] that Skills are even bigger deal than MCP. But I do not see as much hype for Skills as it was for MCP - it seems people are in the MCP "inertia" and having no time to shift to Skills.…

Skills do something you could already do with folder level readme files and hyperlinks inside source, but in a vendor-locked-in way. Not a fan.

Skills are just markdown files in a folder that any agent that can read files can figure out.

Just tell your non-Claude agent to read your skills directory, and extract the preambles.

Re: The Bitter Lesson of LLM Extensions

#42

Post by ttkciar »

bloppe wrote: post #31

How is this related to the bitter lesson?

The author speculates that bigger/smarter models interpreting vague directives to utilize general-function tools will outperform more precise and detailed directives to utilize narrow-function tools:

> Granted to use a skill the agent needs to have general purpose access to a computer, but this is the bitter lesson in action. Giving an agent general purpose tools and trusting it to have the ability to use them to accomplish a task might very well be the winning strategy over making specialized tools for every task.

Re: The Bitter Lesson of LLM Extensions

#43

Post by vidarh »

vessenes wrote: post #3

> "If I could short MCP, I would" I mean, MCP is hard to work with. But there's a very large set of things that we want a hardened interface to out there - if not MCP, it will be something very like it. In particular, MCP was probably overly complicated at the design phase to deal with the realities of streaming text / tokens back and forth live. That is, it chose not to abstract these realities in exchange for some…

The thing is, MCP is little more than another self-descriping API format, and current models can handle most semi-regular API's with just a description and basic tooling. I had Claude interact with my app server via Curl before I decided to just tell it to write an API client instead. I could have told it to implement MCP instead, but now I have a CLI client that I can use as well, and Claude happily uses it with just the --help options.

If you don't already have an API, sure, MCP is a possible choice for that API. But if you have an API, there is decreasing reasons to bother implementing an MPC server the smarter the models are getting vs. just giving it access to your API docs.

Re: The Bitter Lesson of LLM Extensions

#44

Post by ttkciar »

Der_Einzige wrote: post #26

Zero discussion around LLM sampling. How do you leave such a gaping hole in such a written piece? I know it's not AI cus AI wouldn't be that sloppy.

Local inference users are all about sampling, but users addicted to commercial inference services are wary of sampling, because they have to pay by the token.

Re: The Bitter Lesson of LLM Extensions

#46

Post by uriegas »

> "I expect us to go back to extending our agents with the most accessible programming language: natural language."

I don't agree with this. Natural language is so ambiguous. At least for software development the hard work is still coming up with clearly defined solutions. There is a reason for why math has its own domain specific language.

Re: The Bitter Lesson of LLM Extensions

#47

Post by jswny »

ttkciar wrote: post #32

Well, that's just great. The academic community has been using the term "skill" for years, to refer to classes of tasks at which LLMs exhibit competence. Now OpenAI has usurped the term to refer to these inference-guiding .md files. I'm not looking forward to having to pick through a Google hit list for "LLM skills", figuring out which publications are about skills in the traditional sense and which are about the Ope…

Skills are an Anthropic feature

Re: The Bitter Lesson of LLM Extensions

#48

Post by simonw »

lupire wrote: post #34
simonw wrote: post #18

Earlier quoted context omitted.

They're not. They are just a formalization of that pattern, with a very tiny extra feature where the model harness scans that folder on startup and loads some YAML metadata into the system prompt so it knows which ones to read later on.

So "skills" are a hack around the LLM not actually being very smart? Interesting.

Everything we do with LLMs is a hack around them not actually being very smart!

Working around their many limitations has been the nature of the game since the original GPT-3.

Re: The Bitter Lesson of LLM Extensions

#49

Post by conception »

lupire wrote: post #34
simonw wrote: post #18

Earlier quoted context omitted.

They're not. They are just a formalization of that pattern, with a very tiny extra feature where the model harness scans that folder on startup and loads some YAML metadata into the system prompt so it knows which ones to read later on.

So "skills" are a hack around the LLM not actually being very smart? Interesting.

More not wasting context having it figure it out.

It’s documentation vs researching how to do something.

Re: The Bitter Lesson of LLM Extensions

#50

Post by btbuildem »

AIorNot wrote: post #39
WatchDog wrote: post #36

The most useful LLM "extension" isn't even mentioned in this article, and that is shell use. An LLM with a shell integration can do anything you need it to.

A man with a spoon can dig a swimming pool but Id prefer a backhoe

sudo apt-get install backhoe
Post reply on HN