101–110 of 203 posts
Earlier quoted context omitted.
Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?
A Kubernetes for Kubernetes! But what about when we want to upgrade that?
https://github.com/kubernetes-sigs/multi-tenancy/tree/master...
I will die on the hill that almost nobody actually needs kubernetes. You are not Netflix or Google.
Feel free, but it's a bad hill. I can write my own config management, secret management, volume mounting, deployments, replica scaling, hardware affinity, antiaffinity for resilience, rollover kicking out e.g. staging if there's not enough room to schedule production workloads, and the list goes on. I can also figure out how to install and upgrade each piece of software in my universe, across a few different language…
I've had teams create brand new services in hours that used to take a week. They can create, deploy, debug and manage their own service without any help from me. Their service runs and looks exactly vthe same as every other service. When things go wrong, they know where to look.
My team is smaller than it was last year while running 3x as much. That is all from a good combination of CI jobs, terraform and kubernetes. Also, our cloud bill is WAY down from last year because of the fewer number of VMs we need to run.
Is kubernetes a fad when the CTO is happy because costs are down, while productivity and stability are up?
I will die on the hill that almost nobody actually needs kubernetes. You are not Netflix or Google.
Feel free, but it's a bad hill. I can write my own config management, secret management, volume mounting, deployments, replica scaling, hardware affinity, antiaffinity for resilience, rollover kicking out e.g. staging if there's not enough room to schedule production workloads, and the list goes on. I can also figure out how to install and upgrade each piece of software in my universe, across a few different language…
But if you're already in the cloud most of the k8 solutions already exist as individual tools, with little operational maintenance needed. "I need 4GB RAM, 2 cores, I need access to the postgres secret to talk to it, and here is my hostname" is all achievable with lambda's or ecs fargate task definitions and parameter store. Managed services for queues and databases lower operational burden. All of this while not having to risk a cluster upgrade bringing everything down or now having to increase operational overhead by running multiple clusters to have some operational blast doors.
All of this to say if kubernetes is working for you thats great! Congrats. But I'm on the hill with OP. Most platform/infrastructure engineers that I've heard pitch kubernetes because they are trying to solve one problem I.E secrets management or deployment ect... not all of the problems at once and bring in kubernetes complexity when there is already a fully managed, safer and cheaper solution to migrate to doesn't usually make sense.
There's no 100% right or wrong answer but I think you really have to evaluate what you're migrating from when evaluating moving to kubernetes
Earlier quoted context omitted.
https://github.com/kubernetes-sigs/cluster-api
CAPI is for creating clusters only. It doesn't offer any orchestration.
Earlier quoted context omitted.
The tech churn cycle is getting more and more insane. It's the same process repeating endlessly. 1. Identify one problem you want to fix and ignore everything else. 2. Make a tool to manage the problem while still ignoring everything else. 3. Hype the tool up and shove it in every niche and domain possible. 4. Observer how "everything else" bites you in the ass. 5. Identify the worst problem from #4, use it to start…
> Meanwhile, the complexity of our technology goes up and its reliability in practice goes down. I don’t agree with this at all. Reliability in practice is improving at a phenomenal pace. 10 years ago maintenance outages were a normal feature of every service, and unplanned outages were perfectly ordinary occurrences. Consumers today expect a much higher level of availability and reliability, which they receive rathe…
It might take less resource today though to achieve the same, I agree with that.
Earlier quoted context omitted.
> Meanwhile, the complexity of our technology goes up and its reliability in practice goes down. I don’t agree with this at all. Reliability in practice is improving at a phenomenal pace. 10 years ago maintenance outages were a normal feature of every service, and unplanned outages were perfectly ordinary occurrences. Consumers today expect a much higher level of availability and reliability, which they receive rathe…
I don't know which industry you worked in, but outages in the telecom industry was strictly forbidden and came with severe financial penalties even 10 years ago. And those companies managed to adhere to really strict uptime SLAs even then. It might take less resource today though to achieve the same, I agree with that.
10 years ago I was working for a company that provided a financial OLTP service. We had to invest a huge amount of money to be able to provide a reasonable HA architecture, and to be able to meet 4 hour DR SLAs, and we still had weekly maintenance outages. The amount of effort required to accomplish those service levels today is comparatively trivial, and you could reasonably expect even a low-budget one person operation to be able to exceed them.
You’d expect a service outage to be a significant public controversy today for a lot of companies. It’s never been a good thing, but we’ve come a long way from it being a completely routine event for most services. Especially given the explosion in online services.
The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…
Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?
My team runs a fair number of K8s clusters, mostly on Azure AKS. Of the pods in our clusters, a few talk to the AKS API server for their cluster. Those pods that do, will, occasionally, lose contact with the API server. API calls will start timing out. It'll usually resolve on its own after some time, but then come back later. It's somewhat affected by load. (If we reboot a node or nodes, it usually happens after that reboot.)
I think SNAT port exhaustion is likely. Azure claims it isn't, and they don't provide monitoring on it, so I'm forced to take their word on it.
Azure support has been less than helpful. They think we're putting too much load on the API server. The sum total of all of our custom pods that make k8s API calls is only like ~1/3 of the total load. The rest is core k8s components, like the kubelet. That to me, doesn't smell like a lot of load. Their other suggestion is to buy the "Uptime SLA", a contract that adds an SLA to the cluster. Otherwise, all they offer is an SLO. (Which, empirically … we don't get.)
If someone would love to write the rest of this story… I'd love it. We're rolling out the SLA, and it seems promising, but we're hitting what we think is a separate issue. (Sometimes, the API server will refuse connections from particular nodes, usually for good: we've had to reboot the node to unwedge it. This is the second time that's happened; the first time, Azure support told us to they couldn't debug without a live example. Now we have a live example…)
(I do, on the whole, like Kubernetes. It does a lot of things well.)
I absolutely love it when people talk about failures, it is so nice to be able to learn about stuff without having to make the mistake myself.
The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…
Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?
It lets you declaratively define helm deployments, which in turn orchestrate k8s infrastructure.