Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes is hard

rcwz.pl

Post reply on HN
Print view View original

61–70 of 164 posts

Re: Kubernetes is hard

#61

Post by b33j0r »

kazen44 wrote: post #45
sshine wrote: post #42

Earlier quoted context omitted.

> If you're building a cluster from scratch, on your own hardware, setting up the control plane yourself etc. it's very very hard. I'd like to make a distinction: If you do cluster deployment from scratch, Kubernetes actually gets easier. Because you get good at it. What it also gets is more time-consuming. If you aim to replicate all of AWS, then unrealistically so. The art is knowing when to stop. For a lot of peop…

At the end of the day, getting a stable production environment is simply a tradeoff between the amount of complexity you need to make your infrastructure do what you want, and reducing complexity because it removes failure points from the production environment. K8s is nice and all, but if all you really need can be solved by 2 VM's and a way to announce an anycast address (or use a loadbalancer if that is not an opt…

For reasons of experience, all I ever want from a system is that it’s reproducible.

I had a vanity website running k8s in a managed cloud. I thought I was backed up by my provider and original ansible deployment, which was of course developed iteratively.

I originally did this mostly to do a practice deployment, and get the workflow.

A few years later, it went down and I didn’t notice for a few weeks. It was too unimportant to invest in monitoring, and not worth it to do over. Redeploying gave me lots of confusing errors (note: I also do this stuff for work)

Frankly, I was surprised that the provider doing updates would make my years-stable site fall over. I haven’t tried that one again for funsies, yet. It’s the vendor specificity that was my d’oh!

Re: Kubernetes is hard

#62

Post by rglover »

Kubernetes is hard because it's over-complicated and poorly designed. A lot of people don't want to hear that because it was created by The Almighty Google and people have made oodles of money being k8s gurus.

After wasting two years chasing config files, constant deprecations, and a swamp of third-party dependencies that were supposedly "blessed" (all of which led to unnecessary downtime and stress), I swapped it all out with a HAProxy load balancer server in front of some vanilla instances and a few scripts to handle auto-scaling. Since then: I've had zero downtime and scaling is region-specific and chill (and could work up to an infinite number of instances). It just works.

The punchline: just because it's popular, doesn't mean it's the best way to do it.

Re: Kubernetes is hard

#63

Post by YetAnotherNick »

ActorNightly wrote: post #46
overgard wrote: post #31

I agree with the point that production is hard. There's so many things you just don't think about as a developer that end up being important. Log storage, certificate renewal, etc. I think how "hard" kubernetes is depends on how deep you go. If you're building a cluster from scratch, on your own hardware, setting up the control plane yourself etc. it's very very hard. On the other hand, if you're using a hosted servi…

I disagree that production is hard in itself, I think the way people approach production makes it needlessly more difficult. It seems that when launching a service, everyone seems to try to get the entire stack in one shot, which is difficult. A much better approach is "1. Make it work 2. Make it good 3. Make it fast". Your initial prototypes need to define the core functionality, and then you incrementally build stu…

> 1. Make it work 2. Make it good 3. Make it fast

Disagree. The first two objective clashes very much with each other. I have worked in teams that did that. And generally if anything non trivial is done without planning to just "make it work", it would be more often than not, ridden with many design issues which doesn't allow it to be good.

Re: Kubernetes is hard

#64

Post by brody_hamer »

amir734jj wrote: post #25

I would not use k8s unless we are convinced it will benefit us in the long run (think about constant effort that needs to put in to get things running). k8s is not magic. I would just stick with docker-compose or digital ocean for small startup. OR rent a VM on Azure OR if you really really need k8s use a managed k8s.

Docker swarm is a great option too, for production environments. It’s like the production-ready big brother to docker-compose (with better health checks and deployment rollout options). And it has much less of a learning curve than k8s.

Re: Kubernetes is hard

#66

Post by mirekrusin »

endisneigh wrote: post #17

Cost aside, I wonder how far you can get with something like a managed newsql database (Spanner, CockroachDB, Vitess, etc.) and serverless. Most providers at this point offer ephemeral containers or serverless functions. Does a product focused, non infra startup even need k8s? In my honest opinion people should be using Cloud Run. It’s by far Google’s best cloud product. Anyway, going back to the article - k8s is har…

Cloud services are shit unless I can run them locally when developing and testing.

Re: Kubernetes is hard

#67

Post by yourapostasy »

ActorNightly wrote: post #46
overgard wrote: post #31

I agree with the point that production is hard. There's so many things you just don't think about as a developer that end up being important. Log storage, certificate renewal, etc. I think how "hard" kubernetes is depends on how deep you go. If you're building a cluster from scratch, on your own hardware, setting up the control plane yourself etc. it's very very hard. On the other hand, if you're using a hosted servi…

I disagree that production is hard in itself, I think the way people approach production makes it needlessly more difficult. It seems that when launching a service, everyone seems to try to get the entire stack in one shot, which is difficult. A much better approach is "1. Make it work 2. Make it good 3. Make it fast". Your initial prototypes need to define the core functionality, and then you incrementally build stu…

When I see developers use k8s as “programmable infra replacing on-prem functionality as a baseline”, some time after step 3 is, “make it as cheap as on-prem”. Unfortunately, many of the “move fast and break things” tenor that got the services built in k8s in the first place, baked in decisions that will cost as much to re-engineer into an elastic cloud-savvy form that also saves money compared to on-prem.

This is often from organizational or project management mis-coordination than technical choices the developers are fully aware of. Like a system’s owners not prioritizing putting events of interest into a message queue and insisting upon polling for the information on one end. And an offshore operations team saying learning how to support AWS Lambda/Azure Automation/GCP Cloud Functions is an added cost scope to the contract. So the developer is backed into a corner to run cron/SystemD-based polling off a small EC2. Thousands of decisions like this add up fast, and take a major executive committee commitment to root out tech debt and prevent new tech debt from accumulating to unsustainable levels to prevent cloud initiatives from sinking beneath a mire of expensive small decisions in pursuit of new accomplishments. It is a tough balancing act.

Re: Kubernetes is hard

#69

Post by ranger207 »

988747 wrote: post #5
morelisp wrote: post #4

Earlier quoted context omitted.

> Suddenly your program that wasn’t checking for errors, breaks. The memory that you didn’t manage properly becomes now a problem. Yeah, nobody deployed anything and ran it for months, even years, before Kubernetes.

Yes, they did, but did it really take less effort than running it on Kubernetes?

Yes, until you've scaled enough that it wasn't. If you're deploying a dev or staging server or even prod for your first few thousand users then you can get by with a handful of servers and stuff. But a lot of stuff that works well on one or three servers starts working less well on a dozen servers, and it's around that point that the up-front difficulty of k8s starts to pay off with the lower long-term difficulty

Re: Kubernetes is hard

#70

Post by unity1001 »

Turbots wrote: post #57
honkycat wrote: post #47

I am consistently confused by all of the talk about how "hard" Kubernetes is. We spin up EKS. We install the newrelic and datadog log ingestion pods onto it, provided in a nice "helm" format. We install a few other resources via helm, like external secrets, and external dns, and a few others. Kubernetes EKS runs like a champ. My company saves 100k/mo by dynamically scaling our cloud services, all of which are running…

Your company "saves" over 100k/month paying WAY too much for EKS, which is extremely expensive. If you're at any decent scale (looks like you are), then switch to GKE, or switch to on-prem and buy some hardware + a Kubernetes distro like Mirantis/Openshift/Tanzu. Heck, go run k3s on Hetzner and you won't have that much more work, but save literally millions at the scale you're talking about.

> Heck, go run k3s on Hetzner

The moment Hetzner starts providing managed DBs or managed kubernetes, they will wipe the floor with all the big 3...

Post reply on HN