Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

12-factor Agents: Patterns of reliable LLM applications

github.com

Post reply on HN
Print view View original

1–10 of 84 posts

12-factor Agents: Patterns of reliable LLM applications

#1

Post by dhorthy »

I've been building AI agents for a while. After trying every framework out there and talking to many founders building with AI, I've noticed something interesting: most "AI Agents" that make it to production aren't actually that agentic. The best ones are mostly just well-engineered software with LLMs sprinkled in at key points.

So I set out to document what I've learned about building production-grade AI systems: https://github.com/humanlayer/12-factor-agents. It's a set of principles for building LLM-powered software that's reliable enough to put in the hands of production customers.

In the spirit of Heroku's 12 Factor Apps (https://12factor.net/), these principles focus on the engineering practices that make LLM applications more reliable, scalable, and maintainable. Even as models get exponentially more powerful, these core techniques will remain valuable.

I've seen many SaaS builders try to pivot towards AI by building greenfield new projects on agent frameworks, only to find that they couldn't get things past the 70-80% reliability bar with out-of-the-box tools. The ones that did succeed tended to take small, modular concepts from agent building, and incorporate them into their existing product, rather than starting from scratch.

The full guide goes into detail on each principle with examples and patterns to follow. I've seen these practices work well in production systems handling real user traffic.

I'm sharing this as a starting point—the field is moving quickly so these principles will evolve. I welcome your feedback and contributions to help figure out what "production grade" means for AI systems!

12-factor Agents: Patterns of reliable LLM applications
github.com

Re: 12-factor Agents: Patterns of reliable LLM applications

#3

Post by dhorthy »

mertleee wrote: post #2

What are your favorite open source "frameworks" for agents?

i have seen a ton of good ones, and they all have ups and downs. I think rather than focusing on frameworks though, I'm trying to dig into what goes into them, and what's the tradeoff if you try to build most of it yourself instead

but since you asked, to name a few

- ts: mastra, gensx, vercel ai, many others! - python: crew, langgraph, many others!

Re: 12-factor Agents: Patterns of reliable LLM applications

#4

Post by pancsta »

Very informative wiki, thank you, I will definitely use it. So Ive made my own "AI Agents framework" [0] based on actor model, state machines and aspect oriented programming (released just yesterday, no HN post yet) and I really like points 5 and 7:

    5: Unify execution state and business state
    8. Own your control flow
That is exactly what SecAI does, as it's a graph control flow library at it's core (multigraph instead of DAG) and LLM calls are embedded into graph's nodes. The flow is reinforced with negotiation, cancellation and stateful relations, which make it more "organic". Another thing often missed by other frameworks are dedicated devtools (dbg, repl, svg) - programming for failure, inspecting every step in detail, automatic data exporters (metrics, traces, logs, sql), and dead-simple integrations (bash). I've released the first tech demo [1] which showcases all the devtools using a reference implementation of deepresearch (ported from AtomicAgents). You may especially like the Send/Stop button, which is nothings else then "Factor 6. Launch/Pause/Resume with simple APIs". Oh and it's network transparent, so it can scale.

Feel free to reach out.

[0] https://github.com/pancsta/secai

[1] https://youtu.be/0VJzO1S-gV0

Re: 12-factor Agents: Patterns of reliable LLM applications

#5

Post by dhorthy »

pancsta wrote: post #4

Very informative wiki, thank you, I will definitely use it. So Ive made my own "AI Agents framework" [0] based on actor model, state machines and aspect oriented programming (released just yesterday, no HN post yet) and I really like points 5 and 7: 5: Unify execution state and business state 8. Own your control flow That is exactly what SecAI does, as it's a graph control flow library at it's core (multigraph instea…

i like the terminal UI and otel integrations - what tasks are you using this for today?

Re: 12-factor Agents: Patterns of reliable LLM applications

#6

Post by DebtDeflation »

> most "AI Agents" that make it to production aren't actually that agentic. The best ones are mostly just well-engineered software with LLMs sprinkled in at key points

I've been saying that forever, and I think that anyone who actually implements AI in an enterprise context has come to the same conclusion. Using the Anthropic vernacular, AI "workflows" are the solution 90% of the time and AI "agents" maybe 10%. But everyone wants the shiny new object on their CV and the LLM vendors want to bias the market in that direction because running LLMs in a loop drives token consumption through the roof.

Re: 12-factor Agents: Patterns of reliable LLM applications

#7

Post by peab »

DebtDeflation wrote: post #6

> most "AI Agents" that make it to production aren't actually that agentic. The best ones are mostly just well-engineered software with LLMs sprinkled in at key points I've been saying that forever, and I think that anyone who actually implements AI in an enterprise context has come to the same conclusion. Using the Anthropic vernacular, AI "workflows" are the solution 90% of the time and AI "agents" maybe 10%. But e…

I keep trying to tell my PM this

Re: 12-factor Agents: Patterns of reliable LLM applications

#10

Post by [deleted] »

peab wrote: post #7
DebtDeflation wrote: post #6

> most "AI Agents" that make it to production aren't actually that agentic. The best ones are mostly just well-engineered software with LLMs sprinkled in at key points I've been saying that forever, and I think that anyone who actually implements AI in an enterprise context has come to the same conclusion. Using the Anthropic vernacular, AI "workflows" are the solution 90% of the time and AI "agents" maybe 10%. But e…

I keep trying to tell my PM this

[deleted]
Post reply on HN