Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI agents but they're working in big tech

alexsima.substack.com

Post reply on HN
Print view View original

11–20 of 58 posts

Re: AI agents but they're working in big tech

#11

Post by alsima »

tk90 wrote: post #10
simonw wrote: post #3

"Research shows that AI systems with 30+ agents out-performs a simple LLM call in practically any task (see More Agents Is All You Need), reducing hallucinations and improving accuracy." Has anyone heard of that actually playing out practically in real-world applications? This article links to the paper about it - https://arxiv.org/abs/2402.05120 - but I've not heard from anyone who's implementing production systems…

"agent" is a buzzword. All it is, is a bunch of LLM calls in a while loop. - 'rag' is meaningless as a concept. imagine calling a web app 'database augmented programming'. [1] - 'agent' probably just means 'run an llm in a loop' [1] [1] https://x.com/atroyn/status/1819396701217870102

honestly agree. When I first started working with agents I didnt fully understand what it really was either but I eventually fell on a definition of an LLM call that performs a unique function proactively ¯\_(ツ)_/¯.

Re: AI agents but they're working in big tech

#12

Post by alsima »

simonw wrote: post #9
alsima wrote: post #7

Earlier quoted context omitted.

I would check out this company, Swarms ( https://github.com/kyegomez/swarms ) who's working with enterprises to integrate multi-agents. But definitely a great point to focus on, the research paper mentions that the scaling of performance reduces with complexity of the task, which is definitely true for SWE

I totally believe that people are selling solutions around this idea, what I'd like to hear is genuine success stories from people who have used them in production (and aren't currently employed by a vendor).

Hmmm I get what you mean...I think it's hard to sell a solution around this idea, but I think it will become something more like a common practice/performance improvement method. James Huckle on Linkedin (https://www.linkedin.com/feed/update/urn:li:activity:7214295...) mentioned that agent communication would be something more like a hyperparameter to tune which I agree with.

Re: AI agents but they're working in big tech

#13

Post by kgdiem »

simonw wrote: post #9
alsima wrote: post #7

Earlier quoted context omitted.

I would check out this company, Swarms ( https://github.com/kyegomez/swarms ) who's working with enterprises to integrate multi-agents. But definitely a great point to focus on, the research paper mentions that the scaling of performance reduces with complexity of the task, which is definitely true for SWE

I totally believe that people are selling solutions around this idea, what I'd like to hear is genuine success stories from people who have used them in production (and aren't currently employed by a vendor).

Midjourney uses multiple agents for determining if a prompt is appropriate or not.

I kinda did this, too.

I made a 3 agent system — one is a router that parses the request and determines where to send it (to the other 2) one is a chat agent and the third is an image generator.

If the router determines an image is requested, the chat agent is tasked with making a caption to go along with the image.

It works well enough.

Re: AI agents but they're working in big tech

#14

Post by Imnimo »

simonw wrote: post #3

"Research shows that AI systems with 30+ agents out-performs a simple LLM call in practically any task (see More Agents Is All You Need), reducing hallucinations and improving accuracy." Has anyone heard of that actually playing out practically in real-world applications? This article links to the paper about it - https://arxiv.org/abs/2402.05120 - but I've not heard from anyone who's implementing production systems…

I don't even buy that the linked paper justifies the claim. All the paper does is draw multiple samples from an LLM and take the majority vote. They do try integrating their majority vote algorithm with an existing multi-agent system, but it usually performs worse than just straightforwardly asking the model multiple times (see Table 3). I don't understand how the author of this article can make that claim, nor why they believe the linked paper supports it. It does, however, support my prior that LLM agents are snake oil.

Re: AI agents but they're working in big tech

#15

Post by BoredPositron »

kgdiem wrote: post #13
simonw wrote: post #9

Earlier quoted context omitted.

I totally believe that people are selling solutions around this idea, what I'd like to hear is genuine success stories from people who have used them in production (and aren't currently employed by a vendor).

Midjourney uses multiple agents for determining if a prompt is appropriate or not. I kinda did this, too. I made a 3 agent system — one is a router that parses the request and determines where to send it (to the other 2) one is a chat agent and the third is an image generator. If the router determines an image is requested, the chat agent is tasked with making a caption to go along with the image. It works well enoug…

Midjourney does not use an agent system, they use a single call.

Re: AI agents but they're working in big tech

#16

Post by danenania »

aantix wrote: post #4

Is there an agent framework that lives up to the hype? Where you specify a top-level objective, it plans out those objectives, it selects a completion metric so that it knows when to finish, and iterates/reiterates over the output until completion?

> Where you specify a top-level objective, it plans out those objectives, it selects a completion metric so that it knows when to finish, and iterates/reiterates over the output until completion?

I built Plandex[1], which works roughly like this. The goal (so far) is not to take you from an initial prompt to a 100% working solution in one go, but to provide tools that help you iterate your way to a 90-95% solution. You can then fill in the gaps yourself.

I think the idea of a fully autonomous AI engineer is currently mostly hype. Making that the target is good for marketing, but in practice it leads to lots of useless tire-spinning and wasted tokens. It's not a good idea, for example, to have the LLM try to debug its own output by default. It might, on a case-by-case basis, be a good idea to feed an error back to the LLM, but just as often it will be faster for the developer to do the debugging themselves.

1 - https://plandex.ai

Re: AI agents but they're working in big tech

#17

Post by kgdiem »

BoredPositron wrote: post #15
kgdiem wrote: post #13

Earlier quoted context omitted.

Midjourney uses multiple agents for determining if a prompt is appropriate or not. I kinda did this, too. I made a 3 agent system — one is a router that parses the request and determines where to send it (to the other 2) one is a chat agent and the third is an image generator. If the router determines an image is requested, the chat agent is tasked with making a caption to go along with the image. It works well enoug…

Midjourney does not use an agent system, they use a single call.

I remembered reading that they used > 1, found this screengrab of discord on Reddit:

https://www.reddit.com/r/midjourney/comments/137bj1o/new_upd...

Re: AI agents but they're working in big tech

#18

Post by simonw »

kgdiem wrote: post #13
simonw wrote: post #9

Earlier quoted context omitted.

I totally believe that people are selling solutions around this idea, what I'd like to hear is genuine success stories from people who have used them in production (and aren't currently employed by a vendor).

Midjourney uses multiple agents for determining if a prompt is appropriate or not. I kinda did this, too. I made a 3 agent system — one is a router that parses the request and determines where to send it (to the other 2) one is a chat agent and the third is an image generator. If the router determines an image is requested, the chat agent is tasked with making a caption to go along with the image. It works well enoug…

Is that multiple agents, or just multiple prompts? Are agents and prompts the same thing?

Re: AI agents but they're working in big tech

#19

Post by alsima »

Imnimo wrote: post #14
simonw wrote: post #3

"Research shows that AI systems with 30+ agents out-performs a simple LLM call in practically any task (see More Agents Is All You Need), reducing hallucinations and improving accuracy." Has anyone heard of that actually playing out practically in real-world applications? This article links to the paper about it - https://arxiv.org/abs/2402.05120 - but I've not heard from anyone who's implementing production systems…

I don't even buy that the linked paper justifies the claim. All the paper does is draw multiple samples from an LLM and take the majority vote. They do try integrating their majority vote algorithm with an existing multi-agent system, but it usually performs worse than just straightforwardly asking the model multiple times (see Table 3). I don't understand how the author of this article can make that claim, nor why t…

Looking at Table 3: "Our [sampling and voting] method outperforms other methods used standalone in most cases and always enhances other methods across various tasks and LLMs", which benchmark did the majority vote algorithm perform worse in?

Re: AI agents but they're working in big tech

#20

Post by Imnimo »

alsima wrote: post #19
Imnimo wrote: post #14

Earlier quoted context omitted.

I don't even buy that the linked paper justifies the claim. All the paper does is draw multiple samples from an LLM and take the majority vote. They do try integrating their majority vote algorithm with an existing multi-agent system, but it usually performs worse than just straightforwardly asking the model multiple times (see Table 3). I don't understand how the author of this article can make that claim, nor why t…

Looking at Table 3: "Our [sampling and voting] method outperforms other methods used standalone in most cases and always enhances other methods across various tasks and LLMs", which benchmark did the majority vote algorithm perform worse in?

I'm not saying that sampling and majority voting performed worse. I'm saying that multi-agent interaction (labeled Debate and Reflection) performed worse than straightforward approaches that just query multiple times. For example, the Debate method combined with their voting mechanism gets 0.48 GSM8K with Llama2-13B. But majority voting with no multi-agent component (Table 2) gets 0.59 on the same setting. And majority voting with Chain-of-thought (Table 3 CoT/ZS-CoT) does even better.

Fundamentally, drawing multiple independent samples from an LLM is not "AI Agents". The only rows on those tables that are AI Agents are Debate and Reflection. They provide marginal improvements on only a few task/model combinations, and do so at a hefty increase in computational cost. In many tasks, they are significantly behind simpler and cheaper alternatives.

Post reply on HN