Building AI Products–Part I: Back-End Architecture • Hacker News Board

Re: Building AI Products–Part I: Back-End Architecture

#21

Post by pcalcado » Thu, Jan 02, 2025, 3:27 PM UTC

NeutralCrane wrote: post #19

bsenftner wrote: post #11

Earlier quoted context omitted.

Yeah, it's really clear it's an engineer, what the software does is never even mentioned, the purposes and tasks that it performs: what are they? And how is hallucination managed? This reads to me like a complexity soup, where they just started without a clear idea of purpose or goal. Perhaps if the article mentioned what the software does, the purpose, it might be more clear. It sounds like a replacement for the ent…

Agreed. This could be a very intelligent implementation, or it could be an over-engineered mess. It certainly seems like overkill for my experiences with agents, but problem applications can vary wildly. It is impossible to tell how to evaluate these design choices without more concrete details.

This is good feedback; thanks both! Initially, this was a single article, and it started with an explanation of the system, but it was getting too long, so I decided to split it into three. In hindsight, I should have started with part II, where I wanted to talk about the features, but I thought that the most underserved part of the AI stack was the back-end architecture, so I tried to address it first.

Re: Building AI Products–Part I: Back-End Architecture

#22

Post by pcalcado » Thu, Jan 02, 2025, 4:04 PM UTC

tossandthrow wrote: post #10

nikolayasdf123 wrote: post #8

> Agents are not Microservices > Agents naturally align with OOP principles: they maintain encapsulated state (their memory), expose methods (their tools and decision-making capabilities via inference pipelines), and communicate through message passing it does sound like a service (memory=db,methods+messages=api). it is just the level of isolation/deployment you need UPD: also, how come your services share database l…

What I read is that the cut for microservices and agents do not align. This does not mean that agents can not run in microservices, just that is is not 1:1 between agent and a microservice.

Yes, that's what I meant.

There's a whole can of worms here around the "what is a microservice, anyway?" but I tried to avoid more philosophical questions and used the term as shorthand for "small deployable unit following some version of 12 factor for horizontal scalability." It's not super comprehensive but matches what I've seen in practice over the last decade+

Re: Building AI Products–Part I: Back-End Architecture

#23

Post by pcalcado » Thu, Jan 02, 2025, 4:07 PM UTC

nikolayasdf123 wrote: post #8

> Agents are not Microservices > Agents naturally align with OOP principles: they maintain encapsulated state (their memory), expose methods (their tools and decision-making capabilities via inference pipelines), and communicate through message passing it does sound like a service (memory=db,methods+messages=api). it is just the level of isolation/deployment you need UPD: also, how come your services share database l…

By database layer, do you mean the RDS in the diagrams?

If so, they were logical diagrams; the deployment itself was more complicated to handle the realities of AWS and whatnot.

Still, having a single beefy RDS instance is a pretty common pattern for apps at this size. I've never experienced RDS postgres as a bottleneck for standard microservices architectures even at the 100-million-MAU scale.

Re: Building AI Products–Part I: Back-End Architecture

#24

Post by pcalcado » Thu, Jan 02, 2025, 4:08 PM UTC

AYBABTME wrote: post #12

I came to the same conclusion about Temporal for these types of things. Interactive stuff that touches 1 DB? Do it in the API. Needs to coordinate >1 thing? Temporal. Orchestrating a bunch of LLM calls is a perfect fit for Temporal.

If only they embrace Pydantic and the libs Python people actually use >.<

Re: Building AI Products–Part I: Back-End Architecture

#25

Post by pcalcado » Thu, Jan 02, 2025, 4:10 PM UTC

karmasimida wrote: post #7

I don't see AI system too special in terms of back-end engineering, except maybe for agentic system, things are inherently stateful. But considering how limited RPM/TPM with regards mainstream LLMs, states saving/loading is hardly the bottleneck I feel.

Same here. I've had data-intensive systems and classifiers on critical paths for non-AI apps, and the same tools I used before seem to work fine with GenAI.

The primary real difference I've found has to do with when agents make decisions; this creates arbitrary call graphs in your distributed architecture and makes it harder to provision things, optimize, and do anomaly detection.