Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Multi AI agent systems using OpenAI's assistants API

github.com

Post reply on HN
Print view View original

21–30 of 87 posts

Re: Multi AI agent systems using OpenAI's assistants API

#21

Post by purposesystem »

metaskills wrote: post #11
beoberha wrote: post #5

From the website linked in the readme: “A lot of research has been doing in this are and we can expect a lot more in 2024 in this space. I promise to share some clarity around where I think this industry is headed. In personal talks I have warned that multi-agent systems are complex and hard to get right. I've seen little evidence of real-world use cases too” These assistant systems fascinate me, but I just don’t hav…

Thanks @beoberha, I am too. I like one take I heard on Twitter. The sentiment was something like these types of systems are useful under the AI-Powered Productivity industry which has incremental gains, no big bangs. Said another way, if your job was to help a TON of your employees be more productive individually, it is worth it because companies measure those efforts broadly and the payoff is there. But again, not b…

that's actually a great reply, thanks

Re: Multi AI agent systems using OpenAI's assistants API

#22

Post by fzliu »

beoberha wrote: post #5

From the website linked in the readme: “A lot of research has been doing in this are and we can expect a lot more in 2024 in this space. I promise to share some clarity around where I think this industry is headed. In personal talks I have warned that multi-agent systems are complex and hard to get right. I've seen little evidence of real-world use cases too” These assistant systems fascinate me, but I just don’t hav…

A lot of folks I've spoken with say that single-agent systems are still extremely limited, let alone multi-agent platforms. In general, it seems to boil down to:

- Agents need lots of manual tuning and guardrails to make them useful

- Agents with too many guardrails are not general-purpose enough to be worth the time and effort to build

I believe truly great agents will only come from models whose weights are dynamically updated. I hope I'm wrong.

Re: Multi AI agent systems using OpenAI's assistants API

#23

Post by heggy »

metaskills wrote: post #13
phh wrote: post #9

Earlier quoted context omitted.

I've indeed refused to work with some providers giving only a chat interface and not a completion interface because it made the communication "less natural" to the model (like adding new system messages in between for function calling on models which don't officially does it, or adding other categories than system/user/assistant)

Great points. Dont even get me started about how function calling in other LLMs costs me tokens. Something OpenAI provides OOTB. I'm also not a big fan of OpenAI's lock in. Right now I'm on a huge Claude 3 Haiku kick. That said, OpenAI does seem to get the APIs right and my hunch is the new Assistants API is going to potentially disrupt things again. Time will tell.

I would love to be using Claude, but you can't get API access (beyond an initial trial period) in the EU without providing a European VAT number. They don't want personal users or people to even learn and experiment I guess.

Re: Multi AI agent systems using OpenAI's assistants API

#24

Post by pennomi »

dongobread wrote: post #18
beoberha wrote: post #5

From the website linked in the readme: “A lot of research has been doing in this are and we can expect a lot more in 2024 in this space. I promise to share some clarity around where I think this industry is headed. In personal talks I have warned that multi-agent systems are complex and hard to get right. I've seen little evidence of real-world use cases too” These assistant systems fascinate me, but I just don’t hav…

We tried using a multi-agent system for a complex NLP-type task and we found: - Too many errors that just propogate on top of each other, if a single agent in the chain generates something even a little bit off then the whole system goes off the rails. - You often end up having to pass a massive amount of shared context to every agent which just increases the cost dramatically. Curiously enough we had an architect fr…

> Too many errors that just propogate on top of each other

LLMs are like the perfect improv comedy troupe, they virtually always say “yes, and…”

Re: Multi AI agent systems using OpenAI's assistants API

#25

Post by msp26 »

metaskills wrote: post #13
phh wrote: post #9

Earlier quoted context omitted.

I've indeed refused to work with some providers giving only a chat interface and not a completion interface because it made the communication "less natural" to the model (like adding new system messages in between for function calling on models which don't officially does it, or adding other categories than system/user/assistant)

Great points. Dont even get me started about how function calling in other LLMs costs me tokens. Something OpenAI provides OOTB. I'm also not a big fan of OpenAI's lock in. Right now I'm on a huge Claude 3 Haiku kick. That said, OpenAI does seem to get the APIs right and my hunch is the new Assistants API is going to potentially disrupt things again. Time will tell.

> Dont even get me started about how function calling in other LLMs costs me tokens. Something OpenAI provides out of the box.

Not sure what you mean by this.

Re: Multi AI agent systems using OpenAI's assistants API

#26

Post by echelon »

pennomi wrote: post #24
dongobread wrote: post #18

Earlier quoted context omitted.

We tried using a multi-agent system for a complex NLP-type task and we found: - Too many errors that just propogate on top of each other, if a single agent in the chain generates something even a little bit off then the whole system goes off the rails. - You often end up having to pass a massive amount of shared context to every agent which just increases the cost dramatically. Curiously enough we had an architect fr…

> Too many errors that just propogate on top of each other LLMs are like the perfect improv comedy troupe, they virtually always say “yes, and…”

> perfect improv comedy troupe

Check out Vtubers like CodeMiko, who improvs against LLM agents. Or 24/7 streaming LLM cartoon shows that take audience plot suggestions.

Re: Multi AI agent systems using OpenAI's assistants API

#27

Post by behnamoh »

I stay away from such frameworks because:

- Writing what I want in Python/other-lingo gives me much more customizability than these frameworks offer.

- No worries about the future plans of the repo and having to deal with abandonware.

- No vendor lock in. Currently most repos like this focus on OpenAI's models, but I prefer to work with local models of all kinds and any abstraction above llama.cpp or llama-cpp-python is a no-no for me.

The last point means I refuse to build on top of ollama's API as it's yet another wrapper around llama.cpp.

Re: Multi AI agent systems using OpenAI's assistants API

#28

Post by bjterry »

heggy wrote: post #23
metaskills wrote: post #13

Earlier quoted context omitted.

Great points. Dont even get me started about how function calling in other LLMs costs me tokens. Something OpenAI provides OOTB. I'm also not a big fan of OpenAI's lock in. Right now I'm on a huge Claude 3 Haiku kick. That said, OpenAI does seem to get the APIs right and my hunch is the new Assistants API is going to potentially disrupt things again. Time will tell.

I would love to be using Claude, but you can't get API access (beyond an initial trial period) in the EU without providing a European VAT number. They don't want personal users or people to even learn and experiment I guess.

You can use the Claude APIs via OpenRouter with a pre-paid account.

Re: Multi AI agent systems using OpenAI's assistants API

#29

Post by WXLCKNO »

m3kw9 wrote: post #10
beoberha wrote: post #5

From the website linked in the readme: “A lot of research has been doing in this are and we can expect a lot more in 2024 in this space. I promise to share some clarity around where I think this industry is headed. In personal talks I have warned that multi-agent systems are complex and hard to get right. I've seen little evidence of real-world use cases too” These assistant systems fascinate me, but I just don’t hav…

By the time you do get around to it OpenAi would have built a full interface for this. This is the type of stuff that’s gonna get steamrolled.

Pretty much this. I'd love counter examples of startups in the space that haven't been crushed from the top yet.

Re: Multi AI agent systems using OpenAI's assistants API

#30

Post by metaskills »

heggy wrote: post #23
metaskills wrote: post #13

Earlier quoted context omitted.

Great points. Dont even get me started about how function calling in other LLMs costs me tokens. Something OpenAI provides OOTB. I'm also not a big fan of OpenAI's lock in. Right now I'm on a huge Claude 3 Haiku kick. That said, OpenAI does seem to get the APIs right and my hunch is the new Assistants API is going to potentially disrupt things again. Time will tell.

I would love to be using Claude, but you can't get API access (beyond an initial trial period) in the EU without providing a European VAT number. They don't want personal users or people to even learn and experiment I guess.

Interesting, would Amazon Bedrock be an alternative? That's how I use Claude.
Post reply on HN