Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What years of production-grade concurrency teaches us about building AI agents

georgeguimaraes.com

Post reply on HN
Print view View original

31–40 of 53 posts

Re: What years of production-grade concurrency teaches us about building AI agents

#31

Post by zknill »

There's two parts to this article. The scheduler/preemption, and the transport over the network. The article is absolutely right that long-lived request/response over HTTP connections with SSE streamed responses suck.

The article touches very briefly on Phoenix LiveView and Websockets. I wrote about why chatbots hate page refresh[1], and it's not solved by just swapping to Websockets. By far the best mechanism is pub/sub, especially when you can get multi-user/multi-device, conversation hand-off, re-connection, history resumes, and token compaction basically for free from the transport.

1: https://zknill.io/posts/chatbots-worst-enemy-is-page-refresh...

Re: What years of production-grade concurrency teaches us about building AI agents

#32

Post by veunes »

The Let it crash concept is perfect for deterministic bugs, but does it work for probabilistic errors?

If an LLM returns garbage, restarting the process (agent) with the same prompt and temperature 0 yields the same garbage. An Erlang Supervisor restarts a process in a clean state. For an agent "clean state" = lost conversation context

We don't just need Supervision Trees, we need Semantic Supervision Trees that can change strategy on restart. BEAM doesn't give this out of the box, you still code it manually

Re: What years of production-grade concurrency teaches us about building AI agents

#33

Post by veunes »

mackross wrote: post #23

I’m a huge elixir fan, but imho it doesn’t solve durable execution out of the box which is a major problem that often gets swept under the rug by BEAM fanboys. Because ETS and supervision trees don’t play well with deployment via restart, you’ve got to write some level of execution state to relational database or files. You can choose persistent ETS, mnesia, etc, (which have their own tradeoffs and come with some kin…

Spot on. BEAM is great at surviving process crashes, but if the whole cluster goes down or you redeploy, that in-memory state evaporates. It's not magic. For agents that might hang around for days, pure Elixir isn't enough, you still need a persistence layer. The ecosystem is catching up (Oban Pro, FLAME), but in reality, we're still building hybrids: fast actors for active chats and a good old DB for history and long-running processes

Re: What years of production-grade concurrency teaches us about building AI agents

#35

Post by veunes »

simianwords wrote: post #6

I don’t see the point of agent frameworks. Other than durability and checkpoints how does it help me? Claude code already works as an agent that calls tools when necessary so it’s not clear how an abstraction helps here. I have been really confused by langchain and related tech because they seem so bloated without offering me any advantages? I genuinely would like to know what I’m missing.

You don't need frameworks for one-off scripts, but in prod, you're going to need RAG, proper memory, tools, and orchestration anyway. Without standards you'll just end up writing your own janky framework on top of requests. LangChain is definitely a bloated mess, but it provides structure. The beauty of Elixir is that this structure (OTP) is baked into the language, not duct-taped on the side

Re: What years of production-grade concurrency teaches us about building AI agents

#36

Post by wqaatwt »

znnajdla wrote: post #13
mccoyb wrote: post #2

Broadly agree with the author's points, except for this one: > TypeScript/Node.js: Better concurrency story thanks to the event loop, but still fundamentally single-threaded. Worker threads exist but they're heavyweight OS threads, not 2KB processes. There's no preemptive scheduling: one CPU-bound operation blocks everything. This cannot be a real protest: 100% of the time spent in agent frameworks is spent ... waiti…

> Even if you use heavyweight OS threads, I just don't believe this matters. It matters a lot. How many OS threads can you run on 1 machine? With Elixir you can easily run thousands without breaking a sweat. But even if you need only a few agents on one machine, OS thread management is a headache if you have any shared state whatsoever (locks, mutexes, etc.). On Unix you can't even reliably kill dependent processes[1…

Presumably if you can afford to pay for all those tokens the computational cost should be mostly insignificant?

Spending too much time optimizing for the 1% of extra overhead seems suboptimal..

Re: What years of production-grade concurrency teaches us about building AI agents

#37

Post by asa400 »

quadruple wrote: post #27

> The BEAM's "let it crash" philosophy takes the opposite approach. Instead of anticipating every failure mode, you write the happy path and let processes crash. The supervisor detects the crash and restarts the process in a clean state. The rest of the system continues unaffected. Do I want this? If my request fails because the tool doesn't have a DB connection, I want the model to receive information about that err…

You can (and should) always handle whatever errors you actually want to and are able handle, so if that means catching a lot of them and forwarding them to a model, that's not a problem.

The benefit comes mainly from what happens when you encounter unknown errors or errors that you can't handle or errors that would get you into an invalid state. It's normal in BEAM languages to handle the errors you want to/can handle and let the runtime deal with the other transient/unknown errors to restart the process into a known good state.

The big point really is preventing state corruption, so the types of patterns the BEAM encourages will go a long way toward preventing you from accidentally ending up in some kind of unknown zombie state with your model, like for example if your model or control plane think they are connected to each other but actually aren't. That kind of thing.

Happy to clarify more if this sounds strange.

Re: What years of production-grade concurrency teaches us about building AI agents

#38

Post by asa400 »

fud101 wrote: post #26
ipnon wrote: post #11

Earlier quoted context omitted.

Interesting take. I’m an Elixir fanboy because I find LiveView to be very slim. You’re just sending state diffs over WebSocket and updating the DOM with MorphDOM. The simplicity is incomparable to state of the art with JavaScript frameworks in my humble opinion.

I'd use something like a lightweight python framework (take your pick) and pair it with htmx. You can run that on low powered hardware or a cheap VPS. I can't even dev elixir on my N100 minipc, it's too demanding. Otherwise Python and SolidJS or Preact will work perfectly for a SPA.

> I can't even dev elixir on my N100 minipc, it's too demanding.

Isn't the N100 a quad core machine? It can't run Elixir?

Re: What years of production-grade concurrency teaches us about building AI agents

#39

Post by quadruple »

asa400 wrote: post #37
quadruple wrote: post #27

> The BEAM's "let it crash" philosophy takes the opposite approach. Instead of anticipating every failure mode, you write the happy path and let processes crash. The supervisor detects the crash and restarts the process in a clean state. The rest of the system continues unaffected. Do I want this? If my request fails because the tool doesn't have a DB connection, I want the model to receive information about that err…

You can (and should) always handle whatever errors you actually want to and are able handle, so if that means catching a lot of them and forwarding them to a model, that's not a problem. The benefit comes mainly from what happens when you encounter unknown errors or errors that you can't handle or errors that would get you into an invalid state. It's normal in BEAM languages to handle the errors you want to/can handl…

It doesn't sound strange, it actually sounds sane and what everyone should be doing.

At the same time, I can't imagine the last time I had a random exception I didn't think about in prod, but I guess that's the whole point of the BEAM, just don't think about it at all.

I might take a stab at Elixir, the concepts seem interesting and the syntax looks to be up my alley.

Re: What years of production-grade concurrency teaches us about building AI agents

#40

Post by znnajdla »

wqaatwt wrote: post #36
znnajdla wrote: post #13

Earlier quoted context omitted.

> Even if you use heavyweight OS threads, I just don't believe this matters. It matters a lot. How many OS threads can you run on 1 machine? With Elixir you can easily run thousands without breaking a sweat. But even if you need only a few agents on one machine, OS thread management is a headache if you have any shared state whatsoever (locks, mutexes, etc.). On Unix you can't even reliably kill dependent processes[1…

Presumably if you can afford to pay for all those tokens the computational cost should be mostly insignificant? Spending too much time optimizing for the 1% of extra overhead seems suboptimal..

Even if I was building one single agent for a one off hobby project I would still use Elixir. It’s elegantly suited to the job. With any long running failure prone process you’re constantly writing try/catches, health checks, network timeouts, retries, and a whole lot of other orchestration stuff just to keep a flaky real world agent running. With Erlang it’s all just a built in state machine. The process knows what state it is in, crashes, recovers gracefully in exactly the right state.
Post reply on HN