Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

12-factor Agents: Patterns of reliable LLM applications

github.com

Post reply on HN
Print view View original

11–20 of 84 posts

Re: 12-factor Agents: Patterns of reliable LLM applications

#11

Post by film42 »

DebtDeflation wrote: post #6

> most "AI Agents" that make it to production aren't actually that agentic. The best ones are mostly just well-engineered software with LLMs sprinkled in at key points I've been saying that forever, and I think that anyone who actually implements AI in an enterprise context has come to the same conclusion. Using the Anthropic vernacular, AI "workflows" are the solution 90% of the time and AI "agents" maybe 10%. But e…

Everyone wants to go the agent route until the agent messes up once after working 99 times in a row. "Why did it make a silly mistake?" We don't know. "Well, let's put a few more guard rails around it." Sounds good... back to "workflows."

Re: 12-factor Agents: Patterns of reliable LLM applications

#12

Post by film42 »

film42 wrote: post #11
DebtDeflation wrote: post #6

> most "AI Agents" that make it to production aren't actually that agentic. The best ones are mostly just well-engineered software with LLMs sprinkled in at key points I've been saying that forever, and I think that anyone who actually implements AI in an enterprise context has come to the same conclusion. Using the Anthropic vernacular, AI "workflows" are the solution 90% of the time and AI "agents" maybe 10%. But e…

Everyone wants to go the agent route until the agent messes up once after working 99 times in a row. "Why did it make a silly mistake?" We don't know. "Well, let's put a few more guard rails around it." Sounds good... back to "workflows."

"But what about having another agent that quality controls your first agent?"

You should watch the CDO-squared scene from the Big Short again.

Re: 12-factor Agents: Patterns of reliable LLM applications

#13

Post by jlaneve »

mertleee wrote: post #2

What are your favorite open source "frameworks" for agents?

I've been most impressed with Pydantic AI [1], so much so that we ended up building an SDK around it specifically for LLM workflows on Airflow [2].

[1] https://ai.pydantic.dev

[2] https://github.com/astronomer/airflow-ai-sdk

Re: 12-factor Agents: Patterns of reliable LLM applications

#15

Post by dhorthy »

film42 wrote: post #12
film42 wrote: post #11

Earlier quoted context omitted.

Everyone wants to go the agent route until the agent messes up once after working 99 times in a row. "Why did it make a silly mistake?" We don't know. "Well, let's put a few more guard rails around it." Sounds good... back to "workflows."

"But what about having another agent that quality controls your first agent?" You should watch the CDO-squared scene from the Big Short again.

THIS so much. People are like "why human supervision when we can have agent supervsion" and always respond

> look if you don't trust the LLM to make the thing right in the first place, how are you gonna PROBABLY THE SAME LLM to fix it?

yes I know multiple passes improves performance, but it doesn't guarantee anything. for a lot of tool you might wanna call, 90% or even 99% accuracy isn't enough

Re: 12-factor Agents: Patterns of reliable LLM applications

#16

Post by dhorthy »

silasb wrote: post #9

While not specific to 12factor question. With any of these agents and solutions how is LLM Ops being handled? Also, what's the testing strategy and how do I make sure that I don't cause regression?

i try not to take a hard stance on any tool or framework - the idea is take control of the building blocks, and you can still bring most of the cool LLM ops / LLM observability techniques to bear.

I could see one of the twelve factors being around observability beyond just "whats the context" - that may be a good thing to incorporate for version 1.1

Re: 12-factor Agents: Patterns of reliable LLM applications

#17

Post by dhorthy »

film42 wrote: post #11
DebtDeflation wrote: post #6

> most "AI Agents" that make it to production aren't actually that agentic. The best ones are mostly just well-engineered software with LLMs sprinkled in at key points I've been saying that forever, and I think that anyone who actually implements AI in an enterprise context has come to the same conclusion. Using the Anthropic vernacular, AI "workflows" are the solution 90% of the time and AI "agents" maybe 10%. But e…

Everyone wants to go the agent route until the agent messes up once after working 99 times in a row. "Why did it make a silly mistake?" We don't know. "Well, let's put a few more guard rails around it." Sounds good... back to "workflows."

Yup

Re: 12-factor Agents: Patterns of reliable LLM applications

#18

Post by dhorthy »

deadbabe wrote: post #8

With all this AI-agent bullshit out there these days, the most useful AI-agent I still use in daily life is the humble floor vacuum/mopping robot.

They kept telling me automation would do my chores so we could spend more time on writing and art. I write less and still have to do my own laundry
Post reply on HN