Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

blog.stackademic.com

Post reply on HN
Print view View original

111–120 of 216 posts

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#111

Post by astura »

torginus wrote: post #70
figassis wrote: post #18

Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…

Because they have k8s engineers each of whom wants to put it on their resume that they designed and implemented a working cluster in prod.

Resume Driven Development.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#112

Post by justinclift »

woile wrote: post #97
justinclift wrote: post #88

Earlier quoted context omitted.

Are you not using docker swarm behind a reverse proxy? ie nginx has the public facing ipv4 & ipv6 address(es), with docker swarm behind that communicating to the nginx proxies over ipv4

I have traefik as part of the docker-compose file. Installing nginx on the host seems less reproducible, though it could fix my problem. I guess I would choose something as caddy (I'm not that happy with traefik)

I've not used Caddy before personally, but if it does reverse proxying then it'd probably work fine rather than Nginx. :)

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#113

Post by xenospn »

executesorder66 wrote: post #110
xenospn wrote: post #107

I’ve been building software for 25 years across startups and giant megacorps alike, and I still don’t know what Kubernetes is.

And if in the next 25 years you figure out how to use a search engine, you might find out! :p

Maybe one day I’ll have a reason to!

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#114

Post by Havoc »

torton wrote: post #31
Havoc wrote: post #6

How do you end up with 200 yaml file “basic deployments” without anyone looking up from their keyboard and muttering “guys what are we doing”? Honestly they could have picked any stack as next one because the key win here was starting from scratch

“We want to use one standard Helm chart for all applications but then we need it to support all possible variations and use cases across the whole company”

Can’t fix organisational problems with yaml ;)

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#115

Post by ozim »

Havoc wrote: post #6

How do you end up with 200 yaml file “basic deployments” without anyone looking up from their keyboard and muttering “guys what are we doing”? Honestly they could have picked any stack as next one because the key win here was starting from scratch

Every guy that joins or starts new project - instead of reading and getting familiar with what is available does his own stuff.

I see this happening all the time and unless you really have DevOps or SysAdmins who are feel acting like 'assholes' enforcing rules it is going to be like that.

Of course 'assholes' is in quotes because they have to be firm and deny a lot fo crap to keep setup clean - but then also they will be assholes for some that "just want to do stuff".

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#116

Post by mst »

karmarepellent wrote: post #62
oblio wrote: post #59

Earlier quoted context omitted.

Kubernetes upgrades? Don't those risk bricking everything with just 1 environment?

In theory: absolutely. This is just anecdata and you are welcome to challenge me on it, but I have never had a problem upgrading Kubernetes itself. As long as you trail one version behind the latest to ensure critical bugs are fixed before you risk to run into them yourself, I think you are good. Edit: To expand on it a little bit. I think there is always a real, theoretical risk that must be taken into account when…

"I am cutting this corner because I absolutely cannot make a business case I believe in for doing it the hard (but more correct) way but believe me I am still going to be low key paranoid about it indefinitely" is an experience that I think a lot of us can relate to.

I've actually asked for a task to be reassigned to somebody else before now on the grounds that I knew it deserved to be done the simple way but could not for the life of me bring myself to implement that.

(the trick is to find a colleague with a task you *can* do that they hate more and arrange a mutually beneficial swap)

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#117

Post by mst »

edude03 wrote: post #75

Like most tech stories this had pretty much nothing to do the tool itself but with the people/organization. The entire article can be summarized with this one quote > In short, organizational decisions and an overly cautious approach to resource isolation led to an unsustainable number of clusters. And while I emphasize with how they could end up in this situation, it feels like a lot of words were spent blaming the…

The article reads to me as pretty explicitly saying that the only real takeaway wrt k8s itself is "it was the wrong choice for us and then we compounded that wrong choice by making more wrong choices in how we implemented it."

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#119

Post by mst »

whynotmaybe wrote: post #12
risson wrote: post #3

So they made bad architecture decisions, blamed it on Kubernetes for some reason, and then decided to rebuild everything from scratch. Solid. The takeaway being what? Don't make bad decisions?

My personal takeaway, when it fails and you can't blame anyone for any reason, blame a tool. You'll get the whole team helping you when replacing the tool and setting up a better solution. If you blame anyone, people will start to be extra cautious and won't take any initiative. But don't overuse it, if you always blame the tool, you'll end up like my ex colleague "Steve" where every failure was Microsoft's fault.

I've always been fond of blaming myself and asking everybody else to help make sure I don't cock it up a second time - when it works out I get lots of help, lots of useful feedback, and everybody else feels good about putting the effort in.

This does require management who won't punish you for recording it as your fault, though. I've been fairly lucky in that regard.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#120

Post by mrweasel »

rootlocus wrote: post #100
mrweasel wrote: post #96

Earlier quoted context omitted.

> Isn’t the point of kubernetes that you can run your entire infra in a single cluster I've never seen that, but yes 47 seems like a lot. Often you'd need production, staging, test, development, something like that. Then you'd add an additional cluster for running auxiliary service, this is services that has special network access or are not related to you "main product". Maybe a few of these. Still there's a long wa…

> Then you'd add an additional cluster for running auxiliary service, this is services that has special network access or are not related to you "main product". Maybe a few of these. Still there's a long way to 47. Why couldn't you do that with a dedicated node pool, namespaces, taints and affinities? This is how we run our simulators and analytics within the same k8s cluster.

You could do a dedicated node pool and limit the egress to those nodes, but it seems simpler, as in someone is less likely to provision something incorrect, by having a separate cluster.

In my experience companies do not trust Kubernetes to the same extend as they'd trust VLANs and VMs. That's probably not entirely fair, but as you can see from many of the other comments, people find managing Kubernetes extremely difficult to get right.

For some special cases you also have regulatory requirements that maybe could be fulfilled by some Kubernetes combination of node pools, namespacing and so on, but it's not really worth the risk.

From dealing with clients wanting hosted Kubernetes, I can only say that 100% of them have been running multiple clusters. Sometimes for good reason, other times because hosting costs where per project and it's just easier to price out a cluster, compared to buying X% of the capacity on an existing cluster.

One customer I've worked with even ran an entire cluster for a single container, but that was done because no one told the developers to not use that service as an excuse to play with Kubernetes. That was its own kind of stupid.

Post reply on HN