Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What job interviews taught me about Kubernetes

notnotp.com

Post reply on HN
Print view View original

141–150 of 233 posts

Re: What job interviews taught me about Kubernetes

#141

Post by vbezhenar »

mikeocool wrote: post #14

I made this decision at a startup (albeit when the eng team was ~30 people, and we had a monolith with ~10 supporting services). I wouldn’t do it again, even for the reasons stated in the article. The uniformity is nice, we were moving from apps running directly ec2 instances provisioned with ansible. Each time we spun up a new service it was a process to get the ec2 instances provisioned just so. But k8s is such a p…

Docker swarm is that simple solution you're looking for. But people don't need simple solutions. They want scalable solutions and Kubernetes fits this niche perfectly. You can deploy it on single server today and scale to 100 servers managed cluster tomorrow.

Just to provide a similar example. Linux system is insanely complicated. Kernel alone has thousands of options. Distos have tens of thousands of packages. Wherever you look at, everything is hard and complicated. Firewall, containers, init system, filesystem hierarchy, storage layers. One would think that some people desire simpler operating system. But everybody uses Linux despite all these complexities. Try to find OpenBSD in production, for example. It's not easy.

Re: What job interviews taught me about Kubernetes

#142

Post by aranelsurion »

BobbyTables2 wrote: post #29
mikeocool wrote: post #14

I made this decision at a startup (albeit when the eng team was ~30 people, and we had a monolith with ~10 supporting services). I wouldn’t do it again, even for the reasons stated in the article. The uniformity is nice, we were moving from apps running directly ec2 instances provisioned with ansible. Each time we spun up a new service it was a process to get the ec2 instances provisioned just so. But k8s is such a p…

Pretty sure if there was a simple alternative, people would hate it. Everyone initially wants thing A. But then they want to customize it to do all permutations and combinations n of A, B, C. They want it to be extensible. They want redundancy. They want orchestration. They want integration. It’s why practically every config file format eventually becomes its own scripting language. Even HTML started off simple — now…

There are already simpler alternatives, and yes people hate them too. Usually for the opposite reason of k8s: something they need isn’t included, and now bringing it is difficult or impossible.

Fargate and Cloud Run first come to mind.

Re: What job interviews taught me about Kubernetes

#143

Post by kakwa_ »

phailhaus wrote: post #47
reillyse wrote: post #44

The reason this is accelerating recently is agents are really good at spinning up k8s clusters. They've made devops work super super simple. Basically all the annoying stuff you know you should do but it's way too much hassle - using let's encrypt to create unique certs for every app in your cluster to enable zero trust, configuring permissions and security profiles for everything etc etc (never mind just standing th…

The setup may be simple, but what about the maintenance?

And the troubleshooting?

Given the number of moving parts, I would be terrified to have to look under the hood of what Talos deployed for me.

Re: What job interviews taught me about Kubernetes

#144

Post by mijoharas »

Esophagus4 wrote: post #9
clickety_clack wrote: post #6

Adopting k8s when you hire your _second_ engineer (first after the CTO)? That’s a red flag that the CTO’s priorities are wrong and he’s just enjoying tinkering with his infra instead of solving the users’ problems.

I thought that was the point of the article, right? That the tech benefits may not be there, but they’re using it for the non-tech benefits

> That the tech benefits may not be there, but they’re using it for the non-tech benefits

My read of the article is that this is correct, but that the benefits they're using it for are the operational, and organisational.

I think the comment you're replying to is arguing that those benefits don't really matter or outweigh the additional complexity costs when N=2 (engineers). I think I'd probably agree.

Re: What job interviews taught me about Kubernetes

#145

Post by darkwater »

hdjrudni wrote: post #115
perrygeo wrote: post #113

Earlier quoted context omitted.

> Ask your favorite GPT to generate manifests, ... > Before LLMs writing consistent YAMLs was PITA but today on low/development scale it's pretty much free lunch. Writing manifests seems like a trivial thing to focus on. Who operates the k8s cluster in production? Who runs upgrades? Who's on call to monitor the system? Of course if someone else is doing all the work for you, it feels like free lunch!

I find it much easier to upgrade k8s than a bare naked server. With managed k8s, your host upgrades the control plane. And then you can upgrade your PHP, Python, Node, what have you, by flipping a number in your Dockerfile. Not like other forms of sever infra don't need monitoring and upgrades anyway.

The day Kubernetes will have an LTS version supported for 5 years with no API churn, and EKS and other k8s managed systems will have an LTS version based on the k8s LTS plus a bunch of LTS addons supported natively, then I will agree. Actually we will probably live in a better world overall.

Meanwhile, the update stress of core k8s - even managed - is much higher than a good managed old fashioned (immutable) infrastructure.

Re: What job interviews taught me about Kubernetes

#146

Post by darkwater »

clickety_clack wrote: post #53
Esophagus4 wrote: post #9

Earlier quoted context omitted.

I thought that was the point of the article, right? That the tech benefits may not be there, but they’re using it for the non-tech benefits

From the article: > My personal threshold would be the moment the CTO isn't the only engineer anymore. As soon as a second person shows up, the problems K8s solves become real.

What a world we live in with people thinking that the problems of a 2 (founder) engineers startup deserve k8s complexity... Even with LLMs, they are going to rack up tech debt if their focus is - as it should in as mall startup - the final product and not the tech stack itself.

Re: What job interviews taught me about Kubernetes

#147

Post by vbezhenar »

singron wrote: post #99
mikeocool wrote: post #14

I made this decision at a startup (albeit when the eng team was ~30 people, and we had a monolith with ~10 supporting services). I wouldn’t do it again, even for the reasons stated in the article. The uniformity is nice, we were moving from apps running directly ec2 instances provisioned with ansible. Each time we spun up a new service it was a process to get the ec2 instances provisioned just so. But k8s is such a p…

I really wish there was an 80% kubernetes. I think you could get there with some changes: 1. No overlay networks. 1 IP per machine. pods use dynamically allocated ports, and the kubelet enforces pods listen only on their assigned ports using seccomp. 2. No kube-proxy or equivalent Layer-4 "load-balancer". It's not good, but it's often used. You should use some kind of Layer-7 load balancing instead. Also you need to…

1. You can't force third-party software to do that. There are programs with hardcored ports. There are programs which require XML modifications and container rebuilds to change port number. If your platform does not support launching of unmodified containers, it is severely restricted and not suitable for general use. All my programs always use port 8080 for HTTP, I don't make it configurable because I have no reason to.

2. Does not work for all protocols. Again your solution restricts the number of protocols to HTTP protocols. Might work for many uses, but still this restriction doesn't sound very good. Universal load balancer is much simpler conceptually.

3. YAML is not terrible. YAML is awesome. Kubernetes manifests are terrible, that's I agree with. Docker compose is nice, for example. Kubernetes manifests felt like they were designed to be generated from something, but everyone ended up writing them directly or with templates. Though I think that XML generally is superior format so I'd vote for XML in the end.

Overall your suggestions look like you want to shift complexity from cluster operator to software developer. I'm not sure industry supports that, recently it seems to move in the opposite direction, but that's interesting perspective. I guess with some wrappers for some containers it could be made usable.

But honestly you just want to throw away years of progress in containers and network namespaces. I understand that kubernetes mechanisms are somewhat complicated, but the core idea is to make pods look like virtual machines and I think this is very worthy idea.

Re: What job interviews taught me about Kubernetes

#148

Post by vbezhenar »

suralind wrote: post #42

So I’m personally a huge fan of k8s and while I agree it may be „complicated”, it’s because deploying applications is complicated. (I want to point out that there is no requirement no set up cert manager, ArgoCD, external secrets, etc. - and many people who’d consider a VPS would happily slap a .env with an unencrypted secret then ssh to update, but when they choose Kubernetes they take the long route of doing proper…

100% agree with you.

You can actually treat kubernetes as a glorified docker compose engine. Deploy pods, deploy nginx instead of ingress controller, deploy certbot cronjob instead of cert-manager, and believe it or not, it'll work! On a single server!

People often compare Kubernetes with thousands of additional services to a simple VPS, but that's not apples to apples comparison.

Re: What job interviews taught me about Kubernetes

#149

Post by andrewcamel »

I’m not sure it will cause a reversal, but I bet the k8s takeover wouldn’t have happened with costs of today. You inherently end up using far more resources than you need - between HA, built in services, overhead per node, etc.

I personally will be using more resource efficient approaches in everything I do. Question is just what provides the closest set of benefits without the full k8s weight.

Re: What job interviews taught me about Kubernetes

#150

Post by xav0989 »

embedding-shape wrote: post #32
mikeocool wrote: post #24

Earlier quoted context omitted.

Yeah I’ve always meant to check out nomad and never had an opportunity. Though as I recall, it makes heavy use of consul, which I have used in anger, and makes me a little weary (though that experience is likely very out of date).

It doesn't require Consul IIRC, but bunch of features does depend on it, like service discovery and related stuff. But Nomad is totally usable without Consul for simpler setups.

They’ve now had nomad-native services without consul for a while, including health checks!
Post reply on HN