Why microservices when small composable CLI tools seem a better fit for LLMs?
His argument is not about LLM tools but rather about which architecture is better suited for coding with LLMs.
Does coding with LLMs mean more microservices?
11–20 of 73 posts
Re: Does coding with LLMs mean more microservices?
#12A typical rant (composed from memory) goes something like this:
> "These AI types are all delusional. My job is secure. Sure your model can one-shot a small program in green field in 5 minutes with zero debugging. But make it a little larger and it starts to forget features, introduces more bugs than you can fix, and forget letting it loose on large legacy codebases"
What if that's not a diagnosis? What if we see that as an opportunity? O:-)
I'm not saying it needs to be microservices, but say you can constrain the blast radius of an AI going oops (compaction is a famous oops-surface, for instance); and say you can split the work up into self-contained blocks where you can test your i/o and side effects thoroughly...
... well, that's going to be interesting, isn't it?
Programming has always supposed to be about that: Structured programming, functions (preferably side-effect-less for this argument), classes&objects, other forms of modularization including -ok sure- microservices. I'm not sold on exactly the latter because it feels a bit too heavy for me. But ... something like?
Re: Does coding with LLMs mean more microservices?
#13A typical rant (composed from memory) goes something like this: > "These AI types are all delusional. My job is secure. Sure your model can one-shot a small program in green field in 5 minutes with zero debugging. But make it a little larger and it starts to forget features, introduces more bugs than you can fix, and forget letting it loose on large legacy codebases" What if that's not a diagnosis? What if we see tha…
[deleted]
Re: Does coding with LLMs mean more microservices?
#14That's an argument for components with well-defined contracts on their interfaces, but making them microservices just complicates debugging for the model. It's also unclear whether tight coupling is actually a problem when you can refactor this fast.
You are taking the article argument too literally. They meant microservices also in the sense of microlibraries, etc, not strictly a HTTP service.
Re: Does coding with LLMs mean more microservices?
#15[flagged]
Large context windows cost more money. So the pressure is still there to keep it tight.
Re: Does coding with LLMs mean more microservices?
#16[flagged]
Agree on the context window framing. If an LLM needs well-defined boundaries to work well, just write clean module interfaces. You don't need a network boundary for that.
The part about "less scrutiny on PR review" and committing straight to main is telling too. That's not really about microservices, that's just wanting to ship faster with less oversight. Works until it doesn't.
Re: Does coding with LLMs mean more microservices?
#17[dead]
Re: Does coding with LLMs mean more microservices?
#18[flagged]
Agree on the context window framing. If an LLM needs well-defined boundaries to work well, just write clean module interfaces. You don't need a network boundary for that. The part about "less scrutiny on PR review" and committing straight to main is telling too. That's not really about microservices, that's just wanting to ship faster with less oversight. Works until it doesn't.
> The part about "less scrutiny on PR review" and committing straight to main is telling too. That's not really about microservices, that's just wanting to ship faster with less oversight. Works until it doesn't.
And that's the reason I think the author proposes microservices I think. Doesn't need to be microservices, but something where your codebase is split up so that when-not-if it does blow up, you only roll back the one component and try again.
Modularization is hardly a new idea, but might need a slight spin to allow agents to work by themselves a bit more. The speed advantages are too tantalizing not to.
Re: Does coding with LLMs mean more microservices?
#19Earlier quoted context omitted.
Agree on the context window framing. If an LLM needs well-defined boundaries to work well, just write clean module interfaces. You don't need a network boundary for that. The part about "less scrutiny on PR review" and committing straight to main is telling too. That's not really about microservices, that's just wanting to ship faster with less oversight. Works until it doesn't.
> The part about "less scrutiny on PR review" and committing straight to main is telling too. That's not really about microservices, that's just wanting to ship faster with less oversight. Works until it doesn't. And that's the reason I think the author proposes microservices I think. Doesn't need to be microservices, but something where your codebase is split up so that when-not-if it does blow up, you only roll bac…
Expanding:
Think of it this way: A typical sprint in current best practices is 1-2 weeks. Having to scrap a module and start over loses you a lot of time and money. A typical "AI sprint " is A typical rant is "You claim only the output is what counts; but what about the human warmth?". Well, this is IT. If you can thoroughly prove that the inputs and outputs are identical to spec you have done the thing.
Harder than it sounds: CDNs and suss libraries no one told you about, abysmal security, half baked features? Uh.... yeah that happens. But if the blast radius is small, it's fixable and survivable. Hopefully.
Famous last words.
Re: Does coding with LLMs mean more microservices?
#20What matters for LLMs is what matters for humans, which usually means DX. Most Microservice setups are extremely hard to debug across service boundaries, so I think in the future, we'll see more architectural decisions that make sense for LLMs to work with. Which will probably mean modular monoliths or something like that.
Definitively our approach is ai dev ex first.