Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

blog.stackademic.com

Post reply on HN
Print view View original

191–200 of 216 posts

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#191

Post by andix »

fragmede wrote: post #168
andix wrote: post #131

Earlier quoted context omitted.

Isn't one of the strategies also to run one or two backup clusters for any production cluster? Which can take over the workloads if the primary cluster fails for some reason? In a cloud environment the backup cluster can be scaled up quickly if it has to take over, so while it's idling it only requires a few smaller nodes.

You might run a cluster per region, but the whole point of Kubernetes is that it's highly available. What specific piece are you worried about will go down in one cluster that you need two production clusters all the time? Upgrades are a special case where I could see spinning up a backup cluster for.

A lot of things can break (hardware, networking, ...). Spanning the workload over multiple clusters in different regions is already satisfying the "backup cluster" recommendation.

Many workloads don't need to be multi-region as a requirement. So they might run just on one cluster with the option to fail over to another region in case of an emergency. Running a workload on one cluster at a time (even with some downtime for a manual failover) makes a lot of things much easier. Many workloads don't need 99,99% availability, and nothing awful happens if they are down for a few hours.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#192

Post by thephyber »

thephyber wrote: post #92
karmarepellent wrote: post #85

Earlier quoted context omitted.

Do you mind sharing what these operations were? I can think of a few things that may very well brick your control plane. But at the very least existing workloads continue to function in this case as far as I know. Same with e.g. misconfigured network policies. Those might cause downtimes, but at least you can roll them back easily. This was some time ago though. There may be more footguns now. Curious to know how you…

You can’t roll back an AWS EKS control plane version upgrade. “Measure twice, cut once” kinda thing. And operators/helm charts/CRDs use APIs which can and are deprecated, which can cause outages. It pays to make sure your infrastructure is automated with Got apps, CICD, and thorough testing so you can identify the potential hurdles before your cluster upgrade causes unplanned service downtime. It is a huge effort jus…

`s/Got apps/GitOps/`

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#194

Post by m1keil »

marcinzm wrote: post #151
m1keil wrote: post #16

How to save 1M off your cloud infra? Start from a 2M bill. That's how I see most of these projects. You create a massively expensive infra because webscale, then 3 years down the road you (or someone else) gets to rebuild it 10x cheaper. You get to write two blog posts, one for using $tech and one for migrating off $tech. A line in the cv and a promotion. But kudos for them for managing to stop the snowball and actua…

The article says they spend around $150k/year on infra. Given they have 8 DevOps engineers I assume a team of 50+ engineers. Assuming $100k/engineer that's $5 million/year in salary. That's all low end estimates. They saved $100k in the move or 2% of their engineering costs. And they're still very much on the cloud. If you tell most organizations that they need to change everything to save 2% they'll tell you to go a…

I'm not making an argument against the cloud here. Not saying you should move out. The reason why I call out cloud infrastructure specifically is because of how easy it is to let the costs get away from you in the cloud. This is a common thread in every company that uses the cloud. There is a huge amount of waste. And this company's story isn't different.

By the way, 8 DevOps engineers with $150k/year cloud bill deserves to be highlighted here. This is a very high amount of staff dedicated to a relatively small infrastructure setup in an industry that keeps saying "cloud will manage that for you."

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#195

Post by mdaniel »

lenkite wrote: post #190
mdaniel wrote: post #159

Earlier quoted context omitted.

I hate etcd probably more than most, but that 8Gb seems to just be a warning, unless you have information otherwise https://etcd.io/docs/v3.5/dev-guide/limit/#storage-size-limi... I'll take this opportunity to once again bitch and moan that Kubernetes just fucking refuses to allow the KV store to be pluggable, unlike damn near everything else in their world, because they think that's funny or something

It isn't a mere warning. It is strongly recommended as the upper limit. https://www.perfectscale.io/blog/etcd-8gb https://github.com/etcd-io/etcd/issues/9771 And yes, I agree not allowing a pluggable replacement is really stupid.

> https://github.com/etcd-io/etcd/issues/9771

> stale bot marked this as completed (by fucking closing it)

Ah, yes, what would a Kubernetes-adjacent project be without a fucking stale bot to close issues willy nilly

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#196

Post by wyclif »

karmarepellent wrote: post #55
adamtulinius wrote: post #50

Earlier quoted context omitted.

It doesn't take any more competent people to self-host a modern stack, than it does to babysit how a company uses something like Azure. The original value proposition is false, and more and more are realising this.

I think the value proposition holds when you are just getting started with your company and you happen to employ people that know their way around the hyperscaler cloud ecosystems. But I agree that moving your own infra or outsourcing operations when you have managed to do it on your own for a while is most likely misguided. Speaking from experience it introduces costs that cannot possibly be calculdated before the f…

Oh, I've seen this before and it's true in an anecdotal sense for me. One reason why is that they always think of hiring an additional developer as a cost, never savings.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#197

Post by bklw »

The leaps in this writing pain me. There are other aspects, but they’ve been mentioned enough.

Vendor lock in does not come about by relying only on one cloud, but by adopting non-standard technology and interfaces. I do agree that running on multiple providers is the best way of checking if there is lock-in.

Lowering the level of sharing further by running per-service and per-stage clusters, as mentioned in the piece was likewise at best an uninformed decision.

Naturally moving to AWS and letting dedicated teams handle workload orchestration at much higher scale will yield better efficiencies. Ideally without giving up vendor-agnostic deployments by continuing the use of IaC.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#198

Post by _hl_ »

withinboredom wrote: post #174
Elidrake24 wrote: post #134

Earlier quoted context omitted.

Which in many cases would break SOC2 compliance (co-mingling of development and customer resources), and even goes against the basic advice offered in the K8s manual. Beyond that, this limits your ability to test Control Plane upgrades against your stack, though that has generally been very stable in my experience. To be clear I'm not defending the 47 Cluster setup of the OP, just the practice of separating Developme…

Why would you commingle development and customer resources? A k8s cluster is just a control plane, that specifically controls where things are running, and if you specify they can’t share resources, that’s the end of that. If you say they share the same control plane is commingling… then what do you think a cloud console is? And if you are using different accounts there… then I hope you are using dedicated resources…

The frustrating thing with SOC2, or pretty much most compliance requirements, is that they are less about what’s “technically true”, and more about minimizing raised eyebrows.

It does make some sense though. People are not perfect, especially in large organizations, so there is value in just following the masses rather than doing everything your own way.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#199

Post by withinboredom »

_hl_ wrote: post #198
withinboredom wrote: post #174

Earlier quoted context omitted.

Why would you commingle development and customer resources? A k8s cluster is just a control plane, that specifically controls where things are running, and if you specify they can’t share resources, that’s the end of that. If you say they share the same control plane is commingling… then what do you think a cloud console is? And if you are using different accounts there… then I hope you are using dedicated resources…

The frustrating thing with SOC2, or pretty much most compliance requirements, is that they are less about what’s “technically true”, and more about minimizing raised eyebrows. It does make some sense though. People are not perfect, especially in large organizations, so there is value in just following the masses rather than doing everything your own way.

Yes. But it also isn’t a regulation. It is pretty much whatever you say it is.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#200

Post by withinboredom »

AtlasBarfed wrote: post #179
withinboredom wrote: post #178

Earlier quoted context omitted.

This… sounds remarkably like the problems kubernetes solves.

single tank point of failure should be single YAML point of failure mobile autocorrect is super "helpful"

I have completely tanked a kubernetes cluster before. Everything kept working. The only problem was that we couldn’t spin up new containers and if any of the running ones stopped, dns/networking wouldn’t get updated. So for a few hours while we figured out how to fix what I broke, not many issues happened.

So sure, I can kinda see your point, but it feels rather moot. In the cluster, there isnt much that is a single point of failure that also wouldn’t be a point of failure in multiple clusters.

Post reply on HN