Mistral Patent for “Code implemented tool calls”

Regular PSA for patent-related discussions: what matters is the claims, so before commenting one should read those.

Also this is a patent application, so it has not been examined yet. The claims at this point will always be over-broad, because that's just how patent prosecution works: the applicant (brazenly) tries to stake out as much "intellectual territory" as possible (because why not!) but the patent authorities will push back to the extent it does not overlap with other existing "territories" i.e. prior art. The applicant then narrows their claims and the back-and-forth continues until the authority determines that the claims are indeed novel and non-obvious enough to be granted. Every alternate back-and-forth costs $$ and time, which incentivizes applicants to make reasonable claims.

A typical way this negotiation often proceeds is that the applicant starts with a couple of the broadest claims they think reasonable and add multiple narrowing claims (or "dependent" claims) that cover narrower and narrower functionalities, essentially a dependency tree of claims.

This is because the examiner must judge each claim on its own merits, so while the broader claims could get knocked out, the narrower claims could survive and can indicate which "territory" is still unclaimed. The applicant can then focus on that (e.g. by making it part of the root independent claim.)

So often what is interesting in patent applicitations are the dependent claims.

For this patent, the current primary claim literally covers the concept of a tool call by an LLM, and as comments point out, this is ridiculously broad and will get knocked out by prior art. But the dependent claims are intriguing.

A quick glance at those indicates that Mistral thinks the more novel or non-obvious aspects are a "stateless resumable sandbox" which works by executing the generated code until a "non-deterministic operation" (like a random number or a date) and on getting the result of that operation replaying all the code that executed up until that point. The results of previous such operations are cached for reuse in subsequent re-executions.

Now I'm not at all knowledgeable about this area, but I did write a very rudimentary tool-calling LLM harness at one point, and offhand I would now have considered this approach -- re-executing the same code multiple times just seems wasteful and unnecessary. But the specification points out there are other considerations at play in more complex situations, so maybe this is narrower and more interesting than what the title and top claim indicates.

Oh wow :D Not commenting on the larger picture of why or why not Mistral might be doing that, but commenting on the idea itself: I plan to build my own harness soon, and one of its core tenants was, from the beginning, that it would parse every tool call the LLM attempts and executes them itself. I most cases, that would mean to delegate to one of the tools of the rust coreutils, or making a call to a LSP server.

My reasoning: - That way I will have maximum control over how the LLM can interact with the world (i.e. I can even narrow down which arguments I support/not support) - The monitoring story should be great as well, since every interaction with the outside world happens at one place only - I can add functionality as I go, and can understand how the harness changes and why it does what it does from the ground up

The biggest challenge I foresee is with the LLMs making tool calls and having them react to the results. I also wonder how far I will even get with this :D

Back to topic; "code implemented tool calls" is basically my "oh yeah my harness can do tool calls too" (and everything I need is implemented in one codebase, in... code)

Even if it’s good now, three to five years down the line there’s a 50% chance it’s sold for scraps and this may end up in the hands of a patent troll.

Are you saying Mistral didn't invest a lot of money in discovering the invention this patent covers? I'd bet they'd say otherwise. How is the patent office supposed to tell the difference?

I don't think a "pay-to-play" system works. There's already trade secrets that allow corporations to invest lots of money in protecting things that aren't even inventions. You just seem like you're trying to figure which patents are good or bad. But if a patent is widely infringed, by the same kind of logic, it must have tremendous value as an invention.

>Usually it's to defend against another company using a "blocking patent" by pursuing you for infringing their patent which is built upon the thing you patented, doing the next step which was always obvious to you.

Would be pretty hard to sue someone for practicing a patent that cites that person in prior art who concurrently was inventing the same thing and would say how obvious it ones. You're going to tell me it's not so easy, but you completely made up some really narrow and specific situation anyway...

>Not for exclusivity.

It's only your right to exclude others from it that gives a patent any defense... it's all about exclusivity. That's the fundamental nature of property law.