Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building Effective "Agents"

anthropic.com

Post reply on HN
Print view View original

101–110 of 130 posts

Re: Building Effective "Agents"

#101

Post by jcims »

Animats wrote: post #50
simonw wrote: post #11

This is by far the most practical piece of writing I've seen on the subject of "agents" - it includes actionable definitions, then splits most of the value out into "workflows" and describes those in depth with example applications. There's also a cookbook with useful code examples: https://github.com/anthropics/anthropic-cookbook/tree/main/p... Blogged about this here: https://simonwillison.net/2024/Dec/20/building-…

Yes, they have actionable definitions, but they are defining something quite different than the normal definition of an "agent". An agent is a party who acts for another. Often this comes from an employer-employee relationship. This matters mostly when things go wrong. Who's responsible? The airline whose AI agent gave out wrong info about airline policies found, in court, that their "intelligent agent" was considere…

>Anthropic's definition: Some customers define agents as fully autonomous systems that operate independently over extended periods, using various tools to accomplish complex tasks.

But that's not their definition, and they explicitly describe that definition as an 'autonomous system'. Their definition comes in the next paragraph:

"At Anthropic, we categorize all these variations as agentic systems, but draw an important architectural distinction between workflows and agents:

* Workflows are systems where LLMs and tools are orchestrated through predefined code paths. Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks.

* Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks."

Re: Building Effective "Agents"

#102

Post by th0ma5 »

herecomethefuzz wrote: post #100
th0ma5 wrote: post #91

Earlier quoted context omitted.

Probably the least critical and most myth pushing content imo.

> most myth pushing content Care to elaborate?

Lots of lists of the myths of LLMs out there https://masterofcode.com/blog/llms-myths-vs-reality-what-you... Every single post glosses over some aspect of these myths or posits they can be controlled or mitigated in some way, with no examples of anyone else finding applicability of the solutions to real world problems in a supportable and reliable way. When pushed, a myth in the neighborhood of those in the list above is pushed like the system will get better, or some classical computing mechanism will make up the difference, or that the problems aren't so bad, the solution is good enough in some ambiguous way, or that people or existing systems are just as bad when they are not.

Re: Building Effective "Agents"

#103

Post by adeptima »

ldjkfkdsjnv wrote: post #83
adeptima wrote: post #70

The whole Agent thing can easily blow in complexity. Here some challenges I personally faced recently - Durable Execution Paradigm: You may need the system to operate in a "durable execution" fashion like Temporal, Hatchet, Inngest, and Windmill. Your processes need to run for months, be upgraded and restarted. Links below - FSM vs. DAG: Sometimes, a Finite State Machine (FSM) is more appropriate than a Directed Acyc…

Which do you think is the best workflow engine to use here? I've chosen temporal. Engineering management and their background at AWS means the platform is rock solid.

IMHO Temporal and its team is great - it checks all boxes on abstracting away queues, schedulers, distributed state machines for your workflows related load balancers/gateways.

After following discussions and commits Hatchet, Inngest, Windmill, I have a feeling in few years time all other systems will have 95% overlap in core features. They are all influence each other.

Much bigger question what price you will pay by introducing workflow system like Temporal in your code base.

Temporal and co are not for real-time data pubsub.

If latency is an issue or want to keep small memory footprint, better to use something else.

The max payload is 2 MB. It needs to be serializable. Event History has limitations. It's a postgres write heavy.

Bringing the entire team on the same page, and it's not trivial either. If your team has strong Golang developers like in mine. They might oppose it and state something like Temporal is unnecessary abstraction.

Writing your code is fun. Studying and reusing someone else patterns is not so much. Check https://github.com/temporalio/samples-go

For now, I decided to keep prototyping with Temporal and has it running on my personal projects till I create strong use cases and discover all edges.

The great side of effect of exploring Temporal and its competitors you will see better ways of structuring of your code. Especially with distributed state and decoupling execution.

Re: Building Effective "Agents"

#104

Post by simonw »

th0ma5 wrote: post #102
herecomethefuzz wrote: post #100

Earlier quoted context omitted.

> most myth pushing content Care to elaborate?

Lots of lists of the myths of LLMs out there https://masterofcode.com/blog/llms-myths-vs-reality-what-you... Every single post glosses over some aspect of these myths or posits they can be controlled or mitigated in some way, with no examples of anyone else finding applicability of the solutions to real world problems in a supportable and reliable way. When pushed, a myth in the neighborhood of those in the list abov…

I've written extensively about myths and misconceptions about LLMs, much of which overlaps with the observations in that post.

Here's my series about misconceptions: https://simonwillison.net/series/llm-misconceptions/

It doesn't seem to me that you're familiar with my work - you seem to be mixing me in with the vast ocean of uncritical LLM boosting content that's out there.

Re: Building Effective "Agents"

#105

Post by simonw »

Nekit1234007 wrote: post #87
simonw wrote: post #73

Earlier quoted context omitted.

That logic doesn't work for me, because many words have multiple meanings. "Agency" can also be a noun that means an organization that you hire - like a design agency. Or it can mean the CIA. I'm not saying it's not a valid definition of the term, I'm pushing back on the idea that it's THE single correct definition of the term.

May I push back on the idea that a single word may mean (completely) different things?

It's pretty clearly true.

Bank: financial institution, edge of a river, verb to stash something away

Spring: a season, a metal coil, verb to jump

Match: verb to match things together, noun a thing to start fires, noun a competition between two teams

Bat: flying mammal, stick for hitting things

And so on.

Re: Building Effective "Agents"

#107

Post by wahnfrieden »

herecomethefuzz wrote: post #99
temp_praneshp wrote: post #94

Earlier quoted context omitted.

So all you achieved from a spreadsheet's worth of PRs was a 1k LoC reduction?

If you do any software engineering all, you would know that a 1k LoC reduction to achieve the same functionality at the same/better performance is non-trivial.

They’re questioning whether it was a valuable use of time, not whether a spreadsheet of PRs was time-consuming which is apparent

Re: Building Effective "Agents"

#108

Post by OutOfHere »

somnium_sn wrote: post #89
OutOfHere wrote: post #21

Anthropic keeps advertising its MCP (Model Context Protocol), but to the extent it doesn't support other LLMs, e.g. GPT, it couldn't possibly gain adoption. I have yet to see any example of MCP that can be extended to use a random LLM.

You can use it with any LLM in LibreChat, Cody, Zed, etc. See https://modelcontextprotocol.io/clients . The protocol doesn’t prescribe an LLM, has facilities to sample from the client independent of LLM and brings support to build your own bespoke host in their SDK.

Why is it that there isn't a single example showing its use with GPT?

Re: Building Effective "Agents"

#109

Post by dmezzetti »

threecheese wrote: post #86
dmezzetti wrote: post #64

Earlier quoted context omitted.

If you're looking for a lightweight open-source framework designed to handle the patterns mentioned in this article: https://github.com/neuml/txtai Disclaimer: I'm the author of the framework.

Hi David; I’ve seen txtai floating around, and just took a look. Would you say that it fits in a similar niche to something like llamaindex, but starting from a data/embeddings abstraction rather than a retrieval one (building on layers from there - like workflows, agents etc)?

Hello - This is a great and accurate description. The idea is that out of the box there is a pipeline but each component is also customizable.

Re: Building Effective "Agents"

#110

Post by famouswaffles »

th0ma5 wrote: post #93
TobTobXX wrote: post #84

Earlier quoted context omitted.

> How would you trust that the agent is following the criteria, and how sure that the criteria is specific enough? How do you know if a spam filter heuristic works only when intended? You test it. Hard. On the thousands of emails in your archive, on edge-cases you prepare manually, and on the incoming mails. If it doesn't work for some cases, write tests that test for this, adjust prompt and run the test suite. It wo…

This is the location of the arguments. When they work they're "magical" but when they don't work "well people or other things are just as bad" ... And this means that you just cannot reason with the mysticism people have surrounding these things because show stopper problems are minimized or it is implied that they can somehow be reduced. The whole rest of computing does not work this way. So, you can't get the magic…

It's one thing if you're up against the staunch reliability of traditional algorithmic systems but if you're not then it's just silly to hang on it.

You are not getting 100% on email handling whatever method you wish to use. You compare LLMs or probabilistic systems to the best of your alternatives.

There are niches where the lack of complete reliability would be a deal breaker. This isn't one of them and it would be weird to act like it were. People aren't sweeping anything under the rug. It simply just isn't a showstopper.

Post reply on HN