Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: My LLM CLI tool can run tools now, from Python code or plugins

simonwillison.net

Post reply on HN
Print view View original

31–40 of 178 posts

Re: Show HN: My LLM CLI tool can run tools now, from Python code or plugins

#31

Post by setheron »

simonw wrote: post #26
setheron wrote: post #23

Earlier quoted context omitted.

Wow what a great overview; is there a big doc to see all these options? I'd love to try it -- I've been trying `gh` copilot pulgin but this looks more appealing.

I really need to put together a better tutorial - there's a TON of documentation but it's scattered across a bunch of different places: - The official docs: https://llm.datasette.io/ - The workshop I gave at PyCon a few weeks ago: https://building-with-llms-pycon-2025.readthedocs.io/ - The "New releases of LLM" series on my blog: https://simonwillison.net/series/llm-releases/ - My "llm" tag, which has 195 posts now!…

I use NixOS seems like this got me enough to get started (I wanted Gemini)

``` # AI cli (unstable.python3.withPackages ( ps: with ps; [ llm llm-gemini llm-cmd ] )) ```

looks like most of the plugins are models and most of the functionality you demo'd in the parent comment is baked into the tool itself.

Yea a live document might be cool -- part of the interesting bit was seeing "real" type of use cases you use it for .

Anyways will give it a spin.

Re: Show HN: My LLM CLI tool can run tools now, from Python code or plugins

#33

Post by tantalor »

This greatly opens up the risk of footguns.

The doc [1] warns about prompt injection, but I think a more likely scenario is self-inflicted harm. For instance, you give a tool access to your brokerage account to automate trading. Even without prompt injection, there's nothing preventing the bot from making stupid trades.

[1] https://llm.datasette.io/en/stable/tools.html

Re: Show HN: My LLM CLI tool can run tools now, from Python code or plugins

#34

Post by simonw »

ttul wrote: post #32

GPT-4.1 is a capable model, especially for structured outputs and tool calling. I’ve been using LLMs for my day to day grunt work for two years now and this is my goto as a great combination of cheap and capable.

I'm honestly really impressed with GPT-4.1 mini. It is my default from messing around by their API because it is unbelievably inexpensive and genuinely capable at most of the things I throw at it.

I'll switch to o4-mini when I'm writing code, but otherwise 4.1-mini usually does a great job.

Fun example from earlier today:

  llm -f https://raw.githubusercontent.com/BenjaminAster/CSS-Minecraft/refs/heads/main/main.css \
    -s 'explain all the tricks used by this CSS'
That's piping the CSS from that incredible CSS Minecraft demo - https://news.ycombinator.com/item?id=44100148 - into GPT-4.1 mini and asking it for an explanation.

The code is clearly written but entirely uncommented: https://github.com/BenjaminAster/CSS-Minecraft/blob/main/mai...

GPT-4.1 mini's explanation is genuinely excellent: https://gist.github.com/simonw/cafd612b3982e3ad463788dd50287... - it correctly identifies "This CSS uses modern CSS features at an expert level to create a 3D interactive voxel-style UI while minimizing or eliminating JavaScript" and explains a bunch of tricks I hadn't figured out.

And it used 3,813 input tokens and 1,291 output tokens - https://www.llm-prices.com/#it=3813&ot=1291&ic=0.4&oc=1.6 - that's 0.3591 cents (around a third of a cent).

Re: Show HN: My LLM CLI tool can run tools now, from Python code or plugins

#35

Post by abc-1 »

tantalor wrote: post #33

This greatly opens up the risk of footguns. The doc [1] warns about prompt injection, but I think a more likely scenario is self-inflicted harm. For instance, you give a tool access to your brokerage account to automate trading. Even without prompt injection, there's nothing preventing the bot from making stupid trades. [1] https://llm.datasette.io/en/stable/tools.html

[flagged]

Re: Show HN: My LLM CLI tool can run tools now, from Python code or plugins

#36

Post by stavros »

Have you guys had luck with tool calls? I made a simple assistant with access to my calendar, and most models fail to call the tool to add calendar events. GPT-4.1 also regularly tries to gaslight me into believing that it added the event when it didn't call the tool!

Overall, I found tool use extremely hit-and-miss, to the point where I'm sure I'm doing something wrong (I'm using the OpenAI Agents SDK, FWIW).

Re: Show HN: My LLM CLI tool can run tools now, from Python code or plugins

#37

Post by simonw »

stavros wrote: post #36

Have you guys had luck with tool calls? I made a simple assistant with access to my calendar, and most models fail to call the tool to add calendar events. GPT-4.1 also regularly tries to gaslight me into believing that it added the event when it didn't call the tool! Overall, I found tool use extremely hit-and-miss, to the point where I'm sure I'm doing something wrong (I'm using the OpenAI Agents SDK, FWIW).

I get the impression that the key to getting great performance out of tool calls is having a really detailed system prompt, with a bunch of examples.

Anthropic's system prompt just for their "web_search" tool is over 6,000 tokens long! https://simonwillison.net/2025/May/25/claude-4-system-prompt...

Re: Show HN: My LLM CLI tool can run tools now, from Python code or plugins

#38

Post by puttycat »

simonw wrote: post #34
ttul wrote: post #32

GPT-4.1 is a capable model, especially for structured outputs and tool calling. I’ve been using LLMs for my day to day grunt work for two years now and this is my goto as a great combination of cheap and capable.

I'm honestly really impressed with GPT-4.1 mini. It is my default from messing around by their API because it is unbelievably inexpensive and genuinely capable at most of the things I throw at it. I'll switch to o4-mini when I'm writing code, but otherwise 4.1-mini usually does a great job. Fun example from earlier today: llm -f https://raw.githubusercontent.com/BenjaminAster/CSS-Minecraft/refs/heads/main/main.css \…

> while minimizing or eliminating JavaScript

How come it doesn't know for sure?

Re: Show HN: My LLM CLI tool can run tools now, from Python code or plugins

#39

Post by simonw »

tantalor wrote: post #33

This greatly opens up the risk of footguns. The doc [1] warns about prompt injection, but I think a more likely scenario is self-inflicted harm. For instance, you give a tool access to your brokerage account to automate trading. Even without prompt injection, there's nothing preventing the bot from making stupid trades. [1] https://llm.datasette.io/en/stable/tools.html

> This greatly opens up the risk of footguns.

Yeah, it really does.

There are so many ways things can go wrong once you start plugging tools into an LLM, especially if those tool calls are authenticated and can take actions on your behalf.

The MCP world is speed-running this right now, see the GitHub MCP story from yesterday: https://news.ycombinator.com/item?id=44097390

I stuck a big warning in the documentation and I've been careful not to release any initial tool plugins that can cause any damage - hence my QuickJS sandbox one and SQLite plugin being read-only - but it's a dangerous space to be exploring.

(Super fun and fascinating though.)

Re: Show HN: My LLM CLI tool can run tools now, from Python code or plugins

#40

Post by simonw »

puttycat wrote: post #38
simonw wrote: post #34

Earlier quoted context omitted.

I'm honestly really impressed with GPT-4.1 mini. It is my default from messing around by their API because it is unbelievably inexpensive and genuinely capable at most of the things I throw at it. I'll switch to o4-mini when I'm writing code, but otherwise 4.1-mini usually does a great job. Fun example from earlier today: llm -f https://raw.githubusercontent.com/BenjaminAster/CSS-Minecraft/refs/heads/main/main.css \…

> while minimizing or eliminating JavaScript How come it doesn't know for sure?

Because I only showed it the CSS! It doesn't even get the HTML, it's guessed all of that exclusively from what's in the (uncommented) CSS code.

Though it's worth noting that CSS Minecraft was first released three years ago, so there's a chance it has hints about it in the training data already. This is not a meticulous experiment.

(I've had a search around though and the most detailed explanation I could find of how that code works is the one I posted on my blog yesterday - my hunch is that it figured it out from the code alone.)

Post reply on HN