Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes is hard

rcwz.pl

Post reply on HN
Print view View original

31–40 of 164 posts

Re: Kubernetes is hard

#31

Post by overgard »

I agree with the point that production is hard. There's so many things you just don't think about as a developer that end up being important. Log storage, certificate renewal, etc.

I think how "hard" kubernetes is depends on how deep you go. If you're building a cluster from scratch, on your own hardware, setting up the control plane yourself etc. it's very very hard. On the other hand, if you're using a hosted service like EKS and you can hand off the hardware and control plane management to someone else, IMO it's actually very easy to use; I actually find it a lot easier than working with the constellation of services amazon has to offer for instance.

I do think there are parts of it where "best practices" are still being worked out though, like managing YAML files. There's also definitely some rough edges. Like, Helm charts are great... to use. They're an absolute nightmare to write, and there's all sorts of delightful corner cases like not being able to reliably upgrade things that use StatefulSet (last I used anyway). It's not perfect, but honestly if you learn the core concepts and use a hosted service you can get a lot out of it.

Re: Kubernetes is hard

#32

Post by rs999gti »

endisneigh wrote: post #17

Cost aside, I wonder how far you can get with something like a managed newsql database (Spanner, CockroachDB, Vitess, etc.) and serverless. Most providers at this point offer ephemeral containers or serverless functions. Does a product focused, non infra startup even need k8s? In my honest opinion people should be using Cloud Run. It’s by far Google’s best cloud product. Anyway, going back to the article - k8s is har…

> In my honest opinion people should be using Cloud Run. It’s by far Google’s best cloud product.

Is this the same thing as running containers in Azure App Services?

Re: Kubernetes is hard

#33

Post by suralind »

scubbo wrote: post #27

> [K8s] allows dev teams to not worry about all these things; all they must do is to write a simple YAML file. More importantly, teams no longer need to ask DevOps/infra folks to add DNS entry and create a Load Balancer just to expose a service. They can do it on their own, in a declarative manner, if you have an operator do to it. Yeah, as opposed to Cloudformation or Terraform, where you...uhhh... Don't get me wron…

That's true, but I'd argue that TF is not as powerful as k8s. You could combine it with auto scaling services offered by cloud providers, but then you need to understand plugins and with k8s it often is a single value. For example, you can "add" Prometheus scraper just by adding labels. You won't have that with TF.

Re: Kubernetes is hard

#34

Post by majormajor »

suralind wrote: post #30
majormajor wrote: post #23

Earlier quoted context omitted.

Definitely if they weren't running very complex services or their business tolerated an occasional maintenance window. The particular way Kubernetes can bite you is that it makes it much easy to start with far more complex setups - not necessarily much harder than to start with a simple setup! - but then you have to maintain and operate those complex setups forever, even if you don't need them that much. If you're gr…

I think it may appear so because Kubernetes promotes good practices. Do logging, do metrics, do traces. That list quickly grows and while these are good practices, there's a real cost to implement them. But I wouldn't agree that Kubernetes means building tech debt - on the contrary, if you see the tech debt, k8s makes it easier to get rid of it, but that of course takes time and if you don't do it regularly that tech…

I just rarely see people tackling greenfield problems with the discipline to choose to do Kubernetes without also choosing to do "distributed" in a broader, complexity-multiplying way.

If not for Kubernetes (and particularly Kube + cloud offerings) I really doubt they'd do all the setup necessary to get a bunch of distributed systems/services running with such horizontal sprawl.

Re: Kubernetes is hard

#35

Post by solatic »

tptacek wrote: post #14

37signals is not like the typical large-scale startup. They have an extremely small team (around 30 people?), and just a couple of products. Large-scale startups use dynamic-scheduled cloud services in part to reduce coupling between teams. Every service --- and there are dozens --- is scheduled independently, and new teams can get spun up to roll out new services without too much intervention from other teams. When…

> Every service --- and there are dozens...

Most startups that I see trying to go with microservices too early do so while keeping a shared database between the services, so they're not really microservices, but a distributed monolith. This turns into a massive, massive pain.

Doing microservices well means building out templates for CI/CD pipelines, templates for observability, figuring out how best to share (or not share) common credentials like third-party API keys, setting up a service mesh, setting up services like Backstage and Buf... which inevitably requires hiring dedicated engineers at enormous cost.

If you can set up a monolith, why would you switch to microservices? So that you can have a smaller container image and faster autoscaling? So the developers on the second and third teams you hire can wait a little less time for CI tests to run?

It's a pretty big mistake to adopt microservices too early.

Re: Kubernetes is hard

#36

Post by majewsky »

tptacek wrote: post #14

37signals is not like the typical large-scale startup. They have an extremely small team (around 30 people?), and just a couple of products. Large-scale startups use dynamic-scheduled cloud services in part to reduce coupling between teams. Every service --- and there are dozens --- is scheduled independently, and new teams can get spun up to roll out new services without too much intervention from other teams. When…

> Large-scale startups use dynamic-scheduled cloud services in part to reduce coupling between teams.

This is the crux. It's Conway's Law in action. The promise of Kubernetes to a large org is that you can split the baremetal and OS layer into one team that manages everything up to Kubernetes, and then that's the common interface where all other teams deploy their applications into. And besides the team separation, the value is that you have a central layer to put automatic policy enforcement, engineering doctrines, and so forth.

Re: Kubernetes is hard

#37

Post by endisneigh »

rs999gti wrote: post #32
endisneigh wrote: post #17

Cost aside, I wonder how far you can get with something like a managed newsql database (Spanner, CockroachDB, Vitess, etc.) and serverless. Most providers at this point offer ephemeral containers or serverless functions. Does a product focused, non infra startup even need k8s? In my honest opinion people should be using Cloud Run. It’s by far Google’s best cloud product. Anyway, going back to the article - k8s is har…

> In my honest opinion people should be using Cloud Run. It’s by far Google’s best cloud product. Is this the same thing as running containers in Azure App Services?

It’s similar, tho I’d say that is more like App Engine. The azure equivalent is probably Azure container instances

Re: Kubernetes is hard

#38

Post by gautamdivgi »

You need a decent sized team to run an on premise k8s infra. If you’re in the cloud use a managed k8s.

It’s not for everyone in that I agree with the point the author makes. But if you have multiple teams doing app development k8s can be really nice. We do data movement and ML services. AKS has proven great for our use.

Re: Kubernetes is hard

#39

Post by wingmanjd »

znpy wrote: post #9
tkiolp4 wrote: post #2

I think the post title should be called “Production is hard” (as the author talks about later on). Pick up any technology out there: from Python, to C++, to K8s, to Linux… Do the analogous “Hello world” using such technologies and run the program on your laptop. Easy. You congratulate yourself and move on. Production is another story. Suddenly your program that wasn’t checking for errors, breaks. The memory that you…

Quoting Bryan Cantrill: production is war.

I'm a simple man...I see a Bryan Cantrill quote and I upvote.

Re: Kubernetes is hard

#40

Post by joshribakoff »

solatic wrote: post #21
endisneigh wrote: post #17

Cost aside, I wonder how far you can get with something like a managed newsql database (Spanner, CockroachDB, Vitess, etc.) and serverless. Most providers at this point offer ephemeral containers or serverless functions. Does a product focused, non infra startup even need k8s? In my honest opinion people should be using Cloud Run. It’s by far Google’s best cloud product. Anyway, going back to the article - k8s is har…

This. Greenfield products should be serverless by default. By the time you have sustained traffic to the point where you can run the numbers and think that you could save money by switching off serverless, that's a Good Problem To Have, one for which you'll have investors giving you money to hire DevOps to take care of the servers.

Disagree

I tried to use lambda. Cold startup really is awful. You have to deal with running db migrations in step functions or find other solutions. The aurora serverless also does not scale to zero. Once you get traffic you overload RDS and need to pay for and setup a RDS proxy, and dont get me started on the pointless endeavor of trying to keep your lambdas warm. Sort of defeats the point. Serverless is not actually serverless and ends up costing more for less performance and more complexity

Its way simpler and cheaper to start with a single VPS single point of failure, then over time graduate to running docker compose or a single node k3 cluster on that VPS. And then eventually scale out to more nodes…

Post reply on HN