Building AI Products–Part I: Back-End Architecture
philcalcado.com
Building AI Products–Part I: Back-End Architecture
1–10 of 25 posts
Re: Building AI Products–Part I: Back-End Architecture
#2[dead]
Re: Building AI Products–Part I: Back-End Architecture
#3> durable workflows
This is what long-running-transactions of the past became.. and slowly cover all their ground (initially Cadence by Uber, then Temporal). Zillions of little flows that can go through their FSMs at any speed, (milli)seconds-or-days-or-months-or-whenever.
i wonder though, how much some further developments like Cloudflare's durable objects, or similar recently announced Rivet actions [1] would simplify (or, complicate) matters, esp. in this "agentic" case ?
Re: Building AI Products–Part I: Back-End Architecture
#4This article is written by an engineer, first and foremost.
Many of the APIs or LLM extensions provided by AI companies are written by ML engineers that do not have Phil's decades of experience in distributed systems, databases and networking. That is evident after reading this article; the first time I've seen a coherent discussion of the tools and tradeoffs when building agentic systems.
I've struggled to actually build something useful with the "agentic" systems and tools out there (and I've tried a lot). Deep down I've felt intimidated by the dozens of new terms the docs use, and after reflection, those tech marketing pieces give the vibe that they are written primarily by AI and told to be colorful and not clear and precise. These solutions from billion dollar valued companies must to present "brand new" ideas to justify their valuations. We should know better: everything builds on the shoulders of decades of research and discovery. If you see something flying high in the clouds (and not standing on the shoulders of giants), it is sure to fall back to earth soon.
A great read. I'm very excited about Outropy.
Re: Building AI Products–Part I: Back-End Architecture
#5Color me impressed. These guys get it right because they treat LLMs like what they are -- tools with a specific use, not anthropomorphized pets. (although I did groan a bit at the "AI Chief of Staff" moniker).
It's extremely refreshing to hear an actual engineering conversation around LLMs that doesn't sound like it came out of the pages of an undergraduate alchemy notebook.
Re: Building AI Products–Part I: Back-End Architecture
#6Great article, really enjoyed how they described what they initially tried, where it struggled, and why their current solution works better.
Re: Building AI Products–Part I: Back-End Architecture
#7I don't see AI system too special in terms of back-end engineering, except maybe for agentic system, things are inherently stateful.
But considering how limited RPM/TPM with regards mainstream LLMs, states saving/loading is hardly the bottleneck I feel.
Re: Building AI Products–Part I: Back-End Architecture
#8> Agents are not Microservices
> Agents naturally align with OOP principles: they maintain encapsulated state (their memory), expose methods (their tools and decision-making capabilities via inference pipelines), and communicate through message passing
it does sound like a service (memory=db,methods+messages=api). it is just the level of isolation/deployment you need
UPD: also, how come your services share database layer (?), maybe problems in scaling are not due to Agents at all? do you have scaling issues even without agents? would not be surprised! classic rule form Amazon 2002 API mandate by Bezos "no shared db between services. all communication happens over exposed interfaces and over network".
Re: Building AI Products–Part I: Back-End Architecture
#9[deleted]
Re: Building AI Products–Part I: Back-End Architecture
#10> Agents are not Microservices > Agents naturally align with OOP principles: they maintain encapsulated state (their memory), expose methods (their tools and decision-making capabilities via inference pipelines), and communicate through message passing it does sound like a service (memory=db,methods+messages=api). it is just the level of isolation/deployment you need UPD: also, how come your services share database l…
What I read is that the cut for microservices and agents do not align.
This does not mean that agents can not run in microservices, just that is is not 1:1 between agent and a microservice.