Distributed systems programming has stalled

One of the things that is most powerful about K8s is that it gives you a lot of primitives to build things with. This is also its biggest drawback.

If you are running real physical infrastructure and want to run several hundreds of "services" (as in software, not k8s services) then kubernetes is probably a good fit, but you have a storage and secrets problem to solve as well.

On the cloud, unless you're using a managed service, its almost certainly easier to either use lambdas (for low traffic services) or one of the many managed docker hosting services they have.

Some of them are even K8s API compatible.

but _why_?

At its heart, k8s is a "run this thing here with these resources" system. AWS also does this, so duplicating it costs time and money. For most people the benefit of running ~20 services and then k8s permissions) and its an extra layer of things to maintain. For example, running a DB on k8s is perfoectly possible, and there are bunch of patterns you can follow. But you're on the hook for persistence, backup and recovery. managed DBs are more expensive to run, but they cost 0 engineer hours to implement.

BUT

You do get access to helm, which means that you can copypasta mostly working systems into your cluster. (but again like running untrusted docker images, thats not a great thing to do.)

The other thing to note is the networking scheme is badshit crazy and working with ipv6 is still tricky.

1) We are surrounded by distributed systems all the time. When we buy and sell B2B software, we don't know what's stored in our partners databases, they don't know what's in ours. Who should ask whom, and when? If the data sources disagree, whose is correct? Just being given access to a REST API and a couple of webhooks is all you need to be in full distributed systems land.

2) I honestly do not know of a better approach than event-sourcing (i.e. replicated state machine) to coordinate among multiple masters like this. The only technique I can think of that comes close is Paxos - which does not depend on events. But then the first thing I would do if I only had Paxos, would be to use it to bootstrap some kind of event system on top of it.

Even the non-event-sourcing technologies like DBs use events (journals, write-ahead-logs, sstables, etc.) in their own implementation. (However that does not imply that you're getting events 'for free' by using these systems.)

My co-workers do not put any alternatives forward. Reading a database, deciding what action to do, and then carrying out said action is basically the working definition of a race-condition. Bankers and accountants had this figured out thousands of years ago: a bank can't send a wagon across the country with queries like "How much money is in Joe's account?" wait a week for the reply, and then send a second wagon saying "Update Joe's account so it has $36.43 in it now". It's laughable. But now that we have 50-150ms latencies, we feel comfortable doing GETs and POSTs (with a million times more traffic) and somehow think we're not going to get our numbers wrong.

Like, what's an alternative? I have a shiny billion-dollar fully-ACID SQL db with my customer accounts in them. And my SAAS partner bank also has that technology. Put forward literally any idea other than events that will let us coordinate their accounts such that they're not able to double-spend money, or are prevented from spending money if a node is down. I want an alternative to event sourcing.

One of the most significant "triumphs" of my technical career came at a startup where I started as a Principal Engineer and left as the VP Engineering. When I started, we had nightly outages requiring Engineering on-call, and by the time I left, no one could remember a recent issue that required Engineers to wake up.

It was a ton of work and required a strong investment in quality & resilience, but even bigger impact was from observability. We couldn't afford APM, so we took a very deliberate approach to what we logged and how, and stuffed it into an ELK stack for reporting. The immediate benefit was a drastic reduction in time to diagnose issues, and effectively let our small operations team triage issues and easily identify app vs. infra issues almost immediately. Additionally, it was much easier to identify and mitigate fragility in our code and infra.

The net result was an increase in availability from 98.5% to 99.995%, and I think observability contributed to at least half of that.

this would lead to a pointless conversation, if it were to ever happen.