Earlier quoted context omitted.
So there are actually three tech revolutions going on in parallel: 1) Kuberentes 2) Microservices. 3) AI All of them are based on open source, and all of them are permissionless (I.e. the startup does not need permission from a big company, e.g. an app store owner). They also erode the build-in advantages of big tech: 1) Kubernetes erode cloud lock-in on compute/storage. 2) AI is best serve on the edge. I would say t…
Kubernetes is a solution worse than the problem. Microservices are unix pipes but slower. AI is linear algebra that costs x1000 because of the brand name. There is no revolution in software. We are just building crud as high and as fast as we can because the quality of developers tends to zero as the number of developers increases.
I disagree. Maybe it's because we haven't gone "full Kubernetes" where every cronjob is now a Kubernetes cronjob, or what have you, but for all the complexity we've had to overcome, we've seen substantial benefits.
For example, autoscaling instances of my data pipeline apps based on Kafka lag.
Was it complex? Yes. We had to expose Kafka topic lag as a metric to Prometheus, then configure the k8s metrics server that HPAs (Horizontal Pod Autoscalers) use to scale to pull that in from Prometheus using k8s-prometheus-adapter as a custom metric, then set reasonable scaling limits in the data pipeline apps HPAs.
Was it worth it? Fuck yes. We no longer have to worry about data arriving out of time at our data warehouse, because the scaling we've configured (in YAML, god rest its dirty soul) ensures that our data will always arrive at the data warehouse within 5 minutes, which is ideal for a near real time reporting product that greatly enables our end users.
Is Kubernetes worth it? For us, definitely. For anyone else? Really depends on your use case, don't cargo cult this shit.