Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI agents but they're working in big tech

alexsima.substack.com

Post reply on HN
Print view View original

41–50 of 58 posts

Re: AI agents but they're working in big tech

#41

Post by delichon »

tk90 wrote: post #10
simonw wrote: post #3

"Research shows that AI systems with 30+ agents out-performs a simple LLM call in practically any task (see More Agents Is All You Need), reducing hallucinations and improving accuracy." Has anyone heard of that actually playing out practically in real-world applications? This article links to the paper about it - https://arxiv.org/abs/2402.05120 - but I've not heard from anyone who's implementing production systems…

"agent" is a buzzword. All it is, is a bunch of LLM calls in a while loop. - 'rag' is meaningless as a concept. imagine calling a web app 'database augmented programming'. [1] - 'agent' probably just means 'run an llm in a loop' [1] [1] https://x.com/atroyn/status/1819396701217870102

> All it is, is a bunch of LLM calls in a while loop.

Consciousness may be such a loop. Look outward, look at your response, repeat. Like the illusion of motion created by frames per second.

To me it is concerning that proper useful agents may emerge from such a simple loop. E.g just repeatedly asking a sufficiently intelligent AI, "what action within my power shall I perform next in order to maximize the mass of paperclips in the universe," interleaved by performing that act and recording the result.

Re: AI agents but they're working in big tech

#42

Post by refulgentis »

simonw wrote: post #3

"Research shows that AI systems with 30+ agents out-performs a simple LLM call in practically any task (see More Agents Is All You Need), reducing hallucinations and improving accuracy." Has anyone heard of that actually playing out practically in real-world applications? This article links to the paper about it - https://arxiv.org/abs/2402.05120 - but I've not heard from anyone who's implementing production systems…

I spent the last year engineering to the point I could try this and it was ___massively___ disappointing in practice. Massively. Shockingly.

The answers from sampling every big model, then having one distill, were not noticeably better than just from Gpt-4o or Claude Sonnet, and the UX is so much worse (2x wait) that I tabled it for now.

I assumed it would be obviously good, even just from first principles.

I didn't do my usual full med/law benchmarks because given what we saw from a small sample, only 8 questions, I can skip adding it and proceed down the TODO list for launch.

I've also done the inverse, reproduced better results on med with gpt4o x one round RAG x one answer, than Google's Gemini Med insanely complex "finetune our biggest model on med, then do 2 round RAG with 5 answers + a vote on each round, and an opportunity to fetch new documents in round 2". We're both near-saturation, but I got 95% on my random sample of 100 Qs, Med Gemini was 93%.

Re: AI agents but they're working in big tech

#43

Post by bamboozled »

det2x wrote: post #35

It's interesting how long the word "agents"/"intelligent agents" have been around for and how long they've been hyped up for. If you go back to the 80s and 90s you will see how Microsoft was hyping up "intelligent agents" in Windows but nothing ever became of it[1]. I have yet to see an actual useful usecase for agents despite the countless posts asking for examples nobody has provided one. [1] https://www.wired.com/…

I mean they are truly the dream of every capitalist. Intelligent agents basically mean free money. Of course this is something Microsoft would be talking about for decades.

Re: AI agents but they're working in big tech

#44

Post by llm_trw »

simonw wrote: post #3

"Research shows that AI systems with 30+ agents out-performs a simple LLM call in practically any task (see More Agents Is All You Need), reducing hallucinations and improving accuracy." Has anyone heard of that actually playing out practically in real-world applications? This article links to the paper about it - https://arxiv.org/abs/2402.05120 - but I've not heard from anyone who's implementing production systems…

>Has anyone heard of that actually playing out practically in real-world applications?

Yes. I've build tools which do this.

Unfortunately the price tag for building a system that makes agent swarms dynamic is too much for anyone to bear in the current market.

Re: AI agents but they're working in big tech

#45

Post by llm_trw »

tk90 wrote: post #10
simonw wrote: post #3

"Research shows that AI systems with 30+ agents out-performs a simple LLM call in practically any task (see More Agents Is All You Need), reducing hallucinations and improving accuracy." Has anyone heard of that actually playing out practically in real-world applications? This article links to the paper about it - https://arxiv.org/abs/2402.05120 - but I've not heard from anyone who's implementing production systems…

"agent" is a buzzword. All it is, is a bunch of LLM calls in a while loop. - 'rag' is meaningless as a concept. imagine calling a web app 'database augmented programming'. [1] - 'agent' probably just means 'run an llm in a loop' [1] [1] https://x.com/atroyn/status/1819396701217870102

There is a lot more to agents than 'change the system prompt and send it again to the same model'.

It's just that the tools to do this are non-existent and each application needs to be bespoke.

It's like programming without an OS.

Re: AI agents but they're working in big tech

#46

Post by d_sem »

det2x wrote: post #35

It's interesting how long the word "agents"/"intelligent agents" have been around for and how long they've been hyped up for. If you go back to the 80s and 90s you will see how Microsoft was hyping up "intelligent agents" in Windows but nothing ever became of it[1]. I have yet to see an actual useful usecase for agents despite the countless posts asking for examples nobody has provided one. [1] https://www.wired.com/…

It was a common theme in Apple WWDC conferences in the late 80's. Alan Kay has an interesting talk about agents.

I think it could be argued that the low hanging fruit aspect of agents where fulfilled by microservices and web based businesses. The concept of webpages populated with relevant data like google search pages, or Amazon populating products you'd like, could be called agent based. Netflix could be an example of an agent based service.

Re: AI agents but they're working in big tech

#47

Post by aantix »

danenania wrote: post #16
aantix wrote: post #4

Is there an agent framework that lives up to the hype? Where you specify a top-level objective, it plans out those objectives, it selects a completion metric so that it knows when to finish, and iterates/reiterates over the output until completion?

> Where you specify a top-level objective, it plans out those objectives, it selects a completion metric so that it knows when to finish, and iterates/reiterates over the output until completion? I built Plandex[1], which works roughly like this. The goal (so far) is not to take you from an initial prompt to a 100% working solution in one go, but to provide tools that help you iterate your way to a 90-95% solution. Y…

This looked very promising.

Although, it's now prompting me to make an account when I issue `plandex new`?

None of the video demos show this requirement.

I think the demos should show this requirement. Or the Quickstart docs should directly link to the self-hosted instructions.

"? Hey there! It looks like this is your first time using Plandex on this computer.

What would you like to do?

> Start an anonymous trial on Plandex Cloud (no email required)

  Sign in, accept an invite, or create an account"****

Re: AI agents but they're working in big tech

#48

Post by potatoman22 »

simonw wrote: post #3

"Research shows that AI systems with 30+ agents out-performs a simple LLM call in practically any task (see More Agents Is All You Need), reducing hallucinations and improving accuracy." Has anyone heard of that actually playing out practically in real-world applications? This article links to the paper about it - https://arxiv.org/abs/2402.05120 - but I've not heard from anyone who's implementing production systems…

I love this paper though I think their use of "agent" is confusing. My takeaway is ensembling LLM queries is more effective than making a single query. We used a similar approach at work and got a ~30% increase in [performance metric]. It was also cost efficient relative to the ROI increase that came with the performance gains. Simpler systems only require the number of output generations to increase, meaning input costs stay the same regardless of the size of the ensemble.

Re: AI agents but they're working in big tech

#49

Post by potatoman22 »

Imnimo wrote: post #20
alsima wrote: post #19

Earlier quoted context omitted.

Looking at Table 3: "Our [sampling and voting] method outperforms other methods used standalone in most cases and always enhances other methods across various tasks and LLMs", which benchmark did the majority vote algorithm perform worse in?

I'm not saying that sampling and majority voting performed worse. I'm saying that multi-agent interaction (labeled Debate and Reflection) performed worse than straightforward approaches that just query multiple times. For example, the Debate method combined with their voting mechanism gets 0.48 GSM8K with Llama2-13B. But majority voting with no multi-agent component (Table 2) gets 0.59 on the same setting. And majori…

My take: the agents are bad, the ensembling approach is good.

Re: AI agents but they're working in big tech

#50

Post by llm_trw »

simonw wrote: post #18
kgdiem wrote: post #13

Earlier quoted context omitted.

Midjourney uses multiple agents for determining if a prompt is appropriate or not. I kinda did this, too. I made a 3 agent system — one is a router that parses the request and determines where to send it (to the other 2) one is a chat agent and the third is an image generator. If the router determines an image is requested, the chat agent is tasked with making a caption to go along with the image. It works well enoug…

Is that multiple agents, or just multiple prompts? Are agents and prompts the same thing?

Depends on who is trying to sell you what.

Currently all the tools on the market just use a different prompt and call it an agent.

I've build a tool using different models for each agent, e.g. whisper for audio decoding, llava to detect slides on the screen, open cv to crop the image, ocr to read the slide content, llm to summarize everything that's happening during the earnings call.

Post reply on HN