Tools: Code Is All You Need • Hacker News Board

Re: Tools: Code Is All You Need

#121

Post by never_inline » Thu, Jul 03, 2025, 4:59 PM UTC

simonw wrote: post #74

nico wrote: post #73

Earlier quoted context omitted.

> LLM in a sandbox using tools in a loop, you can brute force that problem Does this require using big models through their APIs and spending a lot of tokens? Or can this be done either with local models (probably very slow), or with subscriptions like Claude Code with Pro (without hitting the rate/usage limits)? I saw the Mandelbrot experiment, it was very cool, but still a rather small project, not really comparabl…

The local models aren't quite good enough for this yet in my experience - the big hosted models (o3, Gemini 2.5, Claude 4) only just crossed the capability threshold for this to start working well. I think it's possible we'll see a local model that can do this well within the next few months though - it needs good tool calling, not an encyclopedic knowledge of the world. Might be possible to fit that in a model that…

Wasn't there a tool calling benchmark by docker guys which concluded qwen models are nearly as good as GPT? What is your experience about it?

Personally I am convinced JSON is a bad format for LLMs and code orchestration in python-ish DSL is the future. But local models are pretty bad at code gen too.

Re: Tools: Code Is All You Need

#122

Post by kibwen » Thu, Jul 03, 2025, 4:59 PM UTC

whiplash451 wrote: post #104

kibwen wrote: post #92

Earlier quoted context omitted.

No, 3D printers are the backbone of modern physical prototyping. They're far more important to today's global economy than LLMs are, even if you don't have the vantage point to see it from your sector. That might change in the future, but snapping your fingers to wink LLMs out of existence would change essentially nothing about how the world works today; it would be a non-traumatic non-event. There just hasn't been t…

> snapping your fingers to wink LLMs out of existence would change essentially nothing about how the world works today One could have said the same thing about Google in 2006

No, not even close. By 2006 all sorts of load-bearing infrastructure was relying on Google (e.g. Gmail). Today LLMs are still on the edge of important systems, rather than underlying those systems.

Re: Tools: Code Is All You Need

#123

Post by ramoz » Thu, Jul 03, 2025, 5:00 PM UTC

pramodbiligiri wrote: post #118

Wouldn't the sweet spot for MCP be where the LLM is able to do most of the heavy lifting on its own (outputting some kind of structured or unstructured output), but needs a bit of external/dynamic data that it can't do without? The list of MCP servers/tools it can use should nail that external lookup in a (mostly) deterministic way. This would work best if a human is the end consumer of this output, or will receive m…

You don't solve the problem of being able to rely on the agent to call the MCP.

Hooks into the agent's execution lifecycle seem more reliable for deterministic behavior and supervision.

Re: Tools: Code Is All You Need

#124

Post by jrm4 » Thu, Jul 03, 2025, 5:02 PM UTC

dingnuts wrote: post #59

whiplash451 wrote: post #58

Earlier quoted context omitted.

My reference is the daily usage of chatgpt around me (outside of tech circles). I don’t want to sound like a hard-core LLM believer. I get your point and it’s fair. I just wanted to point out that the current usage of chatgpt is a lot broader than that of 3D printers even at the peak hype of it.

Outside of tech circles it looks like NFTs: people following hype using tech they don't understand which will be popular until the downsides we're aware of that they are ignorant to have consequences, and then the market will reflect the shift in opinion.

Not even remotely in the same universe; the difference is ChatGPT is actually having an impact, people are incorporating it day-to-day in a way that NFTs never stood much of a chance.

Re: Tools: Code Is All You Need

#125

Post by nico » Thu, Jul 03, 2025, 5:02 PM UTC

simonw wrote: post #74

nico wrote: post #73

Earlier quoted context omitted.

> LLM in a sandbox using tools in a loop, you can brute force that problem Does this require using big models through their APIs and spending a lot of tokens? Or can this be done either with local models (probably very slow), or with subscriptions like Claude Code with Pro (without hitting the rate/usage limits)? I saw the Mandelbrot experiment, it was very cool, but still a rather small project, not really comparabl…

The local models aren't quite good enough for this yet in my experience - the big hosted models (o3, Gemini 2.5, Claude 4) only just crossed the capability threshold for this to start working well. I think it's possible we'll see a local model that can do this well within the next few months though - it needs good tool calling, not an encyclopedic knowledge of the world. Might be possible to fit that in a model that…

> it needs good tool calling, not an encyclopedic knowledge of the world

I wonder if there are any groups/companies out there building something like this

Would love to have models that only know 1 or 2 languages (eg. python + js), but are great at them and at tool calling. Definitely don't need my coding agent to know all of Wikipedia and translating between 10 different languages

Re: Tools: Code Is All You Need

#126

Post by tristanz » Thu, Jul 03, 2025, 5:04 PM UTC

JyB wrote: post #114

tristanz wrote: post #84

You can combine MCPs within composable LLM generated code if you put in a little work. At Continual ( https://continual.ai ), we have many workflows that require bulk actions, e.g. iterating over all issues, files, customers, etc. We inject MCP tools into a sandboxed code interpreter and have the agent generate both direct MCP tool calls and composable scripts that leverage MCP tools depending on the task complexity.…

That autocompounding aspect of constantly refining initial prompts with more and more knowledge is so interesting. Gut feeling says it’s something that will be “standardized” in some way, exactly like what MCP did.

Yes, I think you could get quite far with a few tools like memory/todo list + code interpreter + script save/load. You could probably get a lot farther though if you RLVRed this similar to how o3 uses web search so effectively during it's thinking process.

Re: Tools: Code Is All You Need

#127

Post by jrm4 » Thu, Jul 03, 2025, 5:05 PM UTC

Yup, I can't help but think that a lot of the bad thinking comes from trying to avoid the following fact: LLMs are only good where your output does not need to be precise and/or verifiably "perfect," which is kind of the opposite of how code has worked, or has tried to work, in the past.

Right now I got it for: DRAFTS of prose things -- and the only real killer in my opinion, autotagging thousands of old bookmarks. But again, that's just to have cool stuff to go back and peruse, not something that must be correc.t

Re: Tools: Code Is All You Need

#128

Post by never_inline » Thu, Jul 03, 2025, 5:11 PM UTC

The problem I see with MCP is very simple. It's using JSON as the format and that's nowhere as expressive as a programming language.

Consider a python function signature

list_containers(show_stopped: bool = False, name_pattern: Optional[str] = None, sort: Literal["size", "name", "started_at"] = "name"). It doesn't even need docs

Now convert this to JSON schema which is 4x larger input already.

And when generating output, the LLM will generate almost 2x more tokens too, because JSON. Easier to get confused.

And consider that the flow of calling python functions and using their output to call other tools etc... is seen 1000x more times in their fine tuning data, whereas JSON tool calling flows are rare and practically only exist in instruction tuning phase. Then I am sure instruction tuning also contains even more complex code examples where model has to execute complex logic.

Then theres the whole issue of composition. To my knowledge there's no way LLM can do this in one response.

    vehicle = call_func_1()
    if vehicle.type == "car":
      details = lookup_car(vehicle.reg_no)
    else if vehicle.type == "motorcycle":
      details = lookup_motorcycle(vehicle.reg_ni)

How is JSON tool calling going to solve this?

Re: Tools: Code Is All You Need

#129

Post by ivape » Thu, Jul 03, 2025, 5:15 PM UTC

foobarbecue wrote: post #29

pclowes wrote: post #5

Directionally I think this is right. Most LLM usage at scale tends to be filling the gaps between two hardened interfaces. The reliability comes not from the LLM inference and generation but the interfaces themselves only allowing certain configuration to work with them. LLM output is often coerced back into something more deterministic such as types, or DB primary keys. The value of the LLM is determined by how well…

Hype cycle for drones and VR was similar -- at the peak, you have people claiming drones will take over package delivery and everyone will spend their day in VR. Reality is that the applicability is more narrow.

You checked out drone warfare? It’s all the rage in every conflict at the moment. The hype around drones is not fake, and I’d compare it more to autonomous cars because regulation is the only reason you don’t see a million private drones flying around.

Re: Tools: Code Is All You Need

#130

Post by chrisweekly » Thu, Jul 03, 2025, 5:18 PM UTC

never_inline wrote: post #128

The problem I see with MCP is very simple. It's using JSON as the format and that's nowhere as expressive as a programming language. Consider a python function signature list_containers(show_stopped: bool = False, name_pattern: Optional[str] = None, sort: Literal["size", "name", "started_at"] = "name"). It doesn't even need docs Now convert this to JSON schema which is 4x larger input already. And when generating out…

Great point.

But "the" problem with MCP? IMVHO (Very humble, non-expert) the half-baked or missing security aspects are more fundamental. I'd love to hear updates about that from ppl who know what they're talking about.