Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building Effective "Agents"

anthropic.com

Post reply on HN
Print view View original

81–90 of 130 posts

Re: Building Effective "Agents"

#81

Post by mikebelanger »

timdellinger wrote: post #5

My personal view is that the roadmap to AGI requires an LLM acting as a prefrontal cortex: something designed to think about thinking. It would decide what circumstances call for double-checking facts for accuracy, which would hopefully catch hallucinations. It would write its own acceptance criteria for its answers, etc. It's not clear to me how to train each of the sub-models required, or how big (or small!) they n…

> But I think that complex architectures are going to win out over the "just scale up with more data and more compute" approach.

I'm not sure about AGI, but for specialized jobs/tasks (ie having a marketing agent that's familiar with your products and knows how to copywrite for your products) will win over "just add more compute/data" mass-market LLMs. This article does encourage us to keep that architecture simple, which is refreshing to hear. Kind of the AI version of rule of least power.

Admittedly, I have a degree in Cognitive Science, which tended to focus on good 'ol fashioned AI, so I have my biases.

Re: Building Effective "Agents"

#82

Post by jascha_eng »

Kydlaw wrote: post #68
jascha_eng wrote: post #2

I put the agents in quotes because anthropic actually talks more about what they call "workflows". And imo this is where the real value of LLMs currently lies, workflow automation. They also say that using LangChain and other frameworks is mostly unnecessary and does more harm than good. They instead argue to use some simple patterns, directly on the API level. Not dis-similar to the old-school Gang of Four software…

In fact they are mentioning LangGraph (the agent framework from the LangChain company). Imo LangGraph is a much more thoughtful and better built piece of software than the LangChain framework. As I said, they already mention LangGraph in the article, so the Anthropic's conclusions still hold (i.e. KISS). But this thread is going in the wrong direction when talking about LangChain

I'm lumping them all in the same category tbh. They say to just use the model libraries directly or a thin abstraction layer (like litellm maybe?) if you want to keep flexibility to change models easily.

Re: Building Effective "Agents"

#83

Post by ldjkfkdsjnv »

adeptima wrote: post #70

The whole Agent thing can easily blow in complexity. Here some challenges I personally faced recently - Durable Execution Paradigm: You may need the system to operate in a "durable execution" fashion like Temporal, Hatchet, Inngest, and Windmill. Your processes need to run for months, be upgraded and restarted. Links below - FSM vs. DAG: Sometimes, a Finite State Machine (FSM) is more appropriate than a Directed Acyc…

Which do you think is the best workflow engine to use here? I've chosen temporal. Engineering management and their background at AWS means the platform is rock solid.

Re: Building Effective "Agents"

#84

Post by TobTobXX »

skydhash wrote: post #43
ripped_britches wrote: post #38

Earlier quoted context omitted.

Just take any example and think how a human would break it down with decision trees. You are building an AI system to respond to your email. The first agent decides whether the new email should be responded to, yes or no. If no, it can send it to another LLM call that decides to archive it or leave it in the inbox for the human. If yes, it sends it to classifier that decides what type of response is required. Maybe t…

> The first agent decides whether the new email should be responded to, yes or no. How would you trust that the agent is following the criteria, and how sure that the criteria is specific enough. Like someone you just meet told you they going to send you something via email, but then the agent misinterpret it due to missing context and decided to respond in a generic manner leading to misunderstanding. > assume every…

> How would you trust that the agent is following the criteria, and how sure that the criteria is specific enough?

How do you know if a spam filter heuristic works only when intended?

You test it. Hard. On the thousands of emails in your archive, on edge-cases you prepare manually, and on the incoming mails. If it doesn't work for some cases, write tests that test for this, adjust prompt and run the test suite.

It won't ever work in 100% of all cases, but neither do spam filters and we still use them.

Re: Building Effective "Agents"

#85

Post by rybosome »

wahnfrieden wrote: post #77
rybosome wrote: post #20

Earlier quoted context omitted.

I felt deeply vindicated by their assessment of these frameworks, in particular LangChain. I've built and/or worked on a few different LLM-based workflows, and LangChain definitely makes things worse in my opinion. What it boils down to is that we are still coming to understand the right patterns of development for how to develop agents and agentic workflows. LangChain made choices about how to abstract things that a…

The article does not mention the LangChain framework. LangGraph is a different framework, have you tried it?

Yes, our previous lead dev built a lot of our infra using LangGraph. I've been slowly ripping it out since assuming ownership of this part of the codebase.

I've been replacing LangGraph with simple primitives, relying on native Python constructs, etc. For example, instead of building this verbose graph of computation with LangGraph, you can just...call functions in the order you want them. Or declare them async, add them to a list, then await the resolution of all of them.

For a time I was maintaining a spreadsheet of all the refactor PRs, and I had a cumulative reduction of over 1,000 lines of code from these changes. Eventually I stopped keeping track.

^ That's a 1K LOC reduction with no functionality changes. I feel pretty strongly that LangChain/LangGraph are a net negative for our use case.

Re: Building Effective "Agents"

#86

Post by threecheese »

dmezzetti wrote: post #64
simonw wrote: post #11

This is by far the most practical piece of writing I've seen on the subject of "agents" - it includes actionable definitions, then splits most of the value out into "workflows" and describes those in depth with example applications. There's also a cookbook with useful code examples: https://github.com/anthropics/anthropic-cookbook/tree/main/p... Blogged about this here: https://simonwillison.net/2024/Dec/20/building-…

If you're looking for a lightweight open-source framework designed to handle the patterns mentioned in this article: https://github.com/neuml/txtai Disclaimer: I'm the author of the framework.

Hi David; I’ve seen txtai floating around, and just took a look. Would you say that it fits in a similar niche to something like llamaindex, but starting from a data/embeddings abstraction rather than a retrieval one (building on layers from there - like workflows, agents etc)?

Re: Building Effective "Agents"

#87

Post by Nekit1234007 »

simonw wrote: post #73
JonChesterfield wrote: post #67

Earlier quoted context omitted.

Defining "agent" as "thing with agency" seems legitimate to me, what with them being the same word.

That logic doesn't work for me, because many words have multiple meanings. "Agency" can also be a noun that means an organization that you hire - like a design agency. Or it can mean the CIA. I'm not saying it's not a valid definition of the term, I'm pushing back on the idea that it's THE single correct definition of the term.

May I push back on the idea that a single word may mean (completely) different things?

Re: Building Effective "Agents"

#88

Post by threecheese »

zby wrote: post #9
timdellinger wrote: post #5

My personal view is that the roadmap to AGI requires an LLM acting as a prefrontal cortex: something designed to think about thinking. It would decide what circumstances call for double-checking facts for accuracy, which would hopefully catch hallucinations. It would write its own acceptance criteria for its answers, etc. It's not clear to me how to train each of the sub-models required, or how big (or small!) they n…

IMHO with a simple loop LLMs are already capable of some meta thinking, even without any internal new architectures. For me where it still fails is that LLMs cannot catch their own mistakes even some obvious ones. Like with GPT 3.5 I had a persistent problem with the following question: "Who is older, Annie Morton or Terry Richardson?". I was giving it Wikipedia and it was correctly finding out the birth dates of the…

The meta thinking of LLMs is fascinating to me. Here’s a snippet of a convo I had with Claude 3.5 where it struggles with the validity of its own metacognition:

> … true consciousness may require genuine choice or indeterminacy - that is, if an entity's responses are purely deterministic (like a lookup table or pure probability distribution), it might be merely executing a program rather than experiencing consciousness.

> However, even as I articulate this, I face a meta-uncertainty: I cannot know whether my discussion of uncertainty reflects: - A genuine contemplation of these philosophical ideas - A well-trained language model outputting plausible tokens about uncertainty - Some hybrid or different process entirely

> This creates an interesting recursive loop - I'm uncertain about whether my uncertainty is "real" uncertainty or simulated uncertainty. And even this observation about recursive uncertainty could itself be a sophisticated output rather than genuine metacognition.

I actually felt bad for it (him?), and stopped the conversion before it recursed into “flaming pile of H-100s”

Re: Building Effective "Agents"

#89

Post by somnium_sn »

OutOfHere wrote: post #21

Anthropic keeps advertising its MCP (Model Context Protocol), but to the extent it doesn't support other LLMs, e.g. GPT, it couldn't possibly gain adoption. I have yet to see any example of MCP that can be extended to use a random LLM.

You can use it with any LLM in LibreChat, Cody, Zed, etc. See https://modelcontextprotocol.io/clients. The protocol doesn’t prescribe an LLM, has facilities to sample from the client independent of LLM and brings support to build your own bespoke host in their SDK.
Post reply on HN