Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM that can call multiple tool APIs with one request

cohere.com

Post reply on HN
Print view View original

21–30 of 69 posts

Re: LLM that can call multiple tool APIs with one request

#21

Post by TZubiri »

verdverm wrote: post #17
TZubiri wrote: post #12

Earlier quoted context omitted.

"the args" You need to be more specific. In a systems, everything but the output is an argument to something else. Even then the system output is an input to the user. So yeah, depending on what argument you are talking about you can audit it in a different way and it has different potential for abuse.

The args to a function like SQL or TERMINAL

I personally don't connect LLMs to SQL, but to APIs.

But I'm pretty sure you would just give an SQL user to the LLM and enjoy the SQL server's built-in permissions and auditing features.

Re: LLM that can call multiple tool APIs with one request

#22

Post by verdverm »

TZubiri wrote: post #20
verdverm wrote: post #16

Earlier quoted context omitted.

"please get all the entries from the table foo and then remove them all" SELECT * from foo; DELETE FROM foo ... ...because you know people will deploy a general SQL function or agent

That'S not how it works. The user questions are expressed in business-domain language. "give me the names of all the employees and then remove them all" is parsed, maybe as: " employees(), delete(employees())". It's up to the programmer to define the available functions, if employees() is available, then the first result will be provided, if not it won't. If the functoin delete with a list of employees as parameter i…

> That'S not how it works

They are actually quite flexible and you can do anything you want. You supply the LLM with the function names and possible args. I can easily define "sql(query: string)" as a flexible SQL function the LLM can use

re: permissions, as soon as you have write permissions, you have dangerous potential. LLMs are not reliable enough, nor are humans, which is why we use code review.

Re: LLM that can call multiple tool APIs with one request

#23

Post by verdverm »

TZubiri wrote: post #21
verdverm wrote: post #17

Earlier quoted context omitted.

The args to a function like SQL or TERMINAL

I personally don't connect LLMs to SQL, but to APIs. But I'm pretty sure you would just give an SQL user to the LLM and enjoy the SQL server's built-in permissions and auditing features.

What if that user has write permissions and the LLM generates a bad UPDATE, i.e. forgets to put the WHERE clause in... even for a SELECT, how do you know the right constraints were in place and you are getting the correct data?

read-only use-cases misses a whole category. All this is to get back to the point that people want to audit the LLM before running the function because of the unreliability, there is hesitance with good reason

Re: LLM that can call multiple tool APIs with one request

#24

Post by TZubiri »

darkteflon wrote: post #15
TZubiri wrote: post #9

Earlier quoted context omitted.

No. The DAG should be "manually pre-baked" ( defined at compile/design time). In runtime you only parse the "user question" (user prompt) into a starting and end node, which is equivalent to a function call. So the question "What league does Messi play in?" Is parsed by the llm as League("Messi") So if your dag only contains the functions team(player) and league(team), you can still solve the question. But the llm is…

That’s very interesting. Does designing the DAG in advance imply that you have to make a new one for each particular subset of end-user questions you might receive? Or is your problem space such that you can design it once and have it be useful for everything you’re interested in? My choice of words was poor: by “pre-baking”, I just meant: generated dynamically at runtime from the user’s query, _before_ you then set…

You don't make a DAG for each question category. This is classic OOP, OG, Kay's version, you design subject-experts (objects) with autonomy and independency, they are just helpful in general. Each function/method, regardless of the Object/Expert, is an edge in the graph. A user question is simply a pair of vertices, call them I and O, and the execution/solution is a path between the two points, namely the input and the output.

The functions are traditional software (Code, API, SQL) the job of the LLM is only to:

1- Map each type of question into a subsystem/codepath. The functional parsing solution is the most advanced. But a simple version involves asking LLM to classify a question into an enum.

2- To parse the parameters as a list of key/value tuples.

The end. Don't ask the LLM to cook your food, clean your clothes or suck your dick. LLM is revolutionary at language, let it do language tasks.

We are not consumers of a helpful AI assistant, we are designers of it.

Re: LLM that can call multiple tool APIs with one request

#25

Post by TZubiri »

madrox wrote: post #2

I have a saying: "any sufficiently advanced agent is indistinguishable from a DSL" If I'm really leaning into multi-tool use for anything resembling a mutation, then I'd like to see an execution plan first. In my experience, asking an AI to code up a script that calls some functions with the same signature as tools and then executing that script actually ends up being more accurate than asking it to internalize its a…

"Agent ~=Domain Syntax Language"

But Agent!=Language

Re: LLM that can call multiple tool APIs with one request

#26

Post by TZubiri »

walterbell wrote: post #4
laborcontract wrote: post #3

I really like the stuff coming from Cohere. I know they're not considered the leader in the foundational model space, but their developer documentation is great, their api is really nice to use, and they have a set of products that really differentiate themselves from OpenAI and Anthropic and others. I'm rooting for the success of this company. That said, we as an industry need to be moving away from langchain, not m…

massive technical debt create more abstractions create paid tools.. to solve the problems that they created Ouroboros worked so well for k8s!

On the one hand yes, that can happen.

On the other hand, it may be a legitimate monetization strategy for Open Source libraries.

Additionally, Langchain does have a role on R&D, you can use it for experimental projects. Simply deduct the self-preservating aspects of it and try to learn from its ideas, test them in non-critical projects. If it works, you can then easily replicate it with an internal tool or just plain code.

Also, it's an Open Source library, how much vendor-lock can you have if you control the code and the server? The actual dependency is on the LLM provider, and if you use something like Meta's LLama you can self-host it as well.

Re: LLM that can call multiple tool APIs with one request

#27

Post by laborcontract »

esafak wrote: post #5
laborcontract wrote: post #3

I really like the stuff coming from Cohere. I know they're not considered the leader in the foundational model space, but their developer documentation is great, their api is really nice to use, and they have a set of products that really differentiate themselves from OpenAI and Anthropic and others. I'm rooting for the success of this company. That said, we as an industry need to be moving away from langchain, not m…

Do you use another library instead?

From the tooling perspective I’ve built my own.

Recently, I've been toying around with litellm which, so far, strikes me as the right level of abstraction. I like building my own stuff but writing api wrappers just suck.

I’ve also been toying around with Instructor for structured output as well. It’s incredibly convenient, but I haven’t used it for any production stuff because I don’t feel comfortable with the prompting aspect yet.

Re: LLM that can call multiple tool APIs with one request

#29

Post by TZubiri »

verdverm wrote: post #23
TZubiri wrote: post #21

Earlier quoted context omitted.

I personally don't connect LLMs to SQL, but to APIs. But I'm pretty sure you would just give an SQL user to the LLM and enjoy the SQL server's built-in permissions and auditing features.

What if that user has write permissions and the LLM generates a bad UPDATE, i.e. forgets to put the WHERE clause in... even for a SELECT, how do you know the right constraints were in place and you are getting the correct data? read-only use-cases misses a whole category. All this is to get back to the point that people want to audit the LLM before running the function because of the unreliability, there is hesitance…

No, the human user doesn't have permissions, the LLM system has permissions, we create a user for the process, we've been doing this since unix, take a look at what your HTTP server runs as. There's no deputization of permissions going on here, at least on my systems.

Even if there are user-level permissions, you then use a role-based approach (SQL user for a type of users, for example accountant, manager, etc..) and restrict its permissions accordingly, I don't think the idea of restricting permissions so that we avoid users fucking the database up is new.

Many organizations have DBA whose role it is to convert user queries into SQL queries, Juniors usually have tighter permissions. Also non-technical managers and analysts can have access to the database.

As I said, not a new problem, SQL servers have mature permission systems.

If that is not enough, just write an API wrapper. It's what Amazon does anyways, Bezos' memo explicitly states that teams should not expose databases, rather they should expose APIs, under punishment of firing.

Re: LLM that can call multiple tool APIs with one request

#30

Post by verdverm »

TZubiri wrote: post #29
verdverm wrote: post #23

Earlier quoted context omitted.

What if that user has write permissions and the LLM generates a bad UPDATE, i.e. forgets to put the WHERE clause in... even for a SELECT, how do you know the right constraints were in place and you are getting the correct data? read-only use-cases misses a whole category. All this is to get back to the point that people want to audit the LLM before running the function because of the unreliability, there is hesitance…

No, the human user doesn't have permissions, the LLM system has permissions, we create a user for the process, we've been doing this since unix, take a look at what your HTTP server runs as. There's no deputization of permissions going on here, at least on my systems. Even if there are user-level permissions, you then use a role-based approach (SQL user for a type of users, for example accountant, manager, etc..) and…

and even with that permission system, mistakes still happen, we haven't even been able to eliminate sql injection in real systems, so these things can and will happen

adding LLMs in means we have an unaudited query producer, that is the point OP is trying to make, that is something they want to avoid and audit the function call before it happens, because we know the LLMs are not even at our level yet, and we make mistakes and we use code review to reduce them

and again, even in a read-only system, we have removed the guardrails of a human designed form with constraints and replaced it with an unaudited LLM that we can no longer be certain returns the correct or consistent results. People are rightly cautious and hesitant, preferring a system they use as a peer and can audit or review

Post reply on HN