Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes is hard

rcwz.pl

Post reply on HN
Print view View original

101–110 of 164 posts

Re: Kubernetes is hard

#101

Post by dgroshev »

tptacek wrote: post #83
solatic wrote: post #35

Earlier quoted context omitted.

> Every service --- and there are dozens... Most startups that I see trying to go with microservices too early do so while keeping a shared database between the services, so they're not really microservices, but a distributed monolith. This turns into a massive, massive pain. Doing microservices well means building out templates for CI/CD pipelines, templates for observability, figuring out how best to share (or not…

Sure, but I'm not thinking of small startups, but rather large ones like Doordash, Airbnb, Box, Square, Stripe; these places legitimately have lots of services, complex coupling patterns, and a need for some kind of indirection and dynamic placement to enable teams to get things done without coordinating every single PR. 37signals presumably doesn't have any of those problems. They should be using K8s. In the previou…

You are talking about a handful of unicorns, not just large startups, and even then only a subset of them — Shopify with their widely successful monolith comes to mind. By definition their requirements in their current unicorn state can't be the default.

Yet we collectively behave as they are and everyone needs dynamic scheduling (and the rest of the kitchen sink).

Re: Kubernetes is hard

#103

Post by Rantenki »

0xbadcafebee wrote: post #55

> Kubernetes is complex and I think they are partially right Kubernetes is a distributed centralized operating system which itself depends on a distributed decentralized database, and has a varying network topology, permissions system, plugins, scheduler, storage, and much more, depending on how & where it was built, and runs applications as independent containerized environments (often deeply dependent on Linux kern…

OMFG so much this.

I worked at a large company that deployed it's own Kubernetes stack, on a VERY large number of physical hosts. The theory was that the K8S would simplify our devops story enough that we could iterate quickly and scale linearly.

In reality, the K8S team ended up being literally 10x larger than the team building the application we were deploying on it. In addition, K8S introduced entirely new categories of failure mode (ahem; CNI updates/restarts/bedsh*tting, operator/custom resource failures, and tons of other ego driven footguns).

The worst part? The application itself ran fine on a single dev workstation, but also on any random assortment of VMs. Just pass the consul details as environment variables. I am not saying everybody on K8S is in the same boat, but I think that far more people are planning on becoming a unicorn cloud service than have any hope of becoming a unicorn cloud service.

TL;DR: If your hosting solution requires more maintenance than the application itself, you made a boo-boo.

Re: Kubernetes is hard

#104

Post by preseinger »

xyzzy_plugh wrote: post #88
tptacek wrote: post #14

37signals is not like the typical large-scale startup. They have an extremely small team (around 30 people?), and just a couple of products. Large-scale startups use dynamic-scheduled cloud services in part to reduce coupling between teams. Every service --- and there are dozens --- is scheduled independently, and new teams can get spun up to roll out new services without too much intervention from other teams. When…

Who said anything about a large-scale startup? Kubernetes is approachable all the way down to N=1 employees. I strongly disagree with your take on static vs dynamic scheduling. Static scheduling ties your hands early. In a mature organization, it is very much an optimization. Dynamic scheduling forces a cattle-not-pets mentality out of the gate, which is great. It also gives you all the knobs to figure out HA, right-…

deploying kubernetes in an org of O(1) or O(10) or even arguably O(100) engineers is professional negligence

the benefits just don't outweigh the costs, until you get to org sizes where dunbar's number starts to matter

"static scheduling" is the right choice for ~80% of technology orgs

Re: Kubernetes is hard

#106

Post by CharlieDigital »

mirekrusin wrote: post #66
endisneigh wrote: post #17

Cost aside, I wonder how far you can get with something like a managed newsql database (Spanner, CockroachDB, Vitess, etc.) and serverless. Most providers at this point offer ephemeral containers or serverless functions. Does a product focused, non infra startup even need k8s? In my honest opinion people should be using Cloud Run. It’s by far Google’s best cloud product. Anyway, going back to the article - k8s is har…

Cloud services are shit unless I can run them locally when developing and testing.

Google's Firebase emulator is really, really good. Fantastic first party emulator suite.

LocalStack for AWS also let's you emulate a huge chunk of AWS locally. I use it to emulate SNS/SQS locally.

Re: Kubernetes is hard

#107

Post by CharlieDigital »

rs999gti wrote: post #32
endisneigh wrote: post #17

Cost aside, I wonder how far you can get with something like a managed newsql database (Spanner, CockroachDB, Vitess, etc.) and serverless. Most providers at this point offer ephemeral containers or serverless functions. Does a product focused, non infra startup even need k8s? In my honest opinion people should be using Cloud Run. It’s by far Google’s best cloud product. Anyway, going back to the article - k8s is har…

> In my honest opinion people should be using Cloud Run. It’s by far Google’s best cloud product. Is this the same thing as running containers in Azure App Services?

Azure Container Apps is the equivalent.

AWS also has App Runner but it doesn't scale to 0.

Re: Kubernetes is hard

#108

Post by crabmusket »

tptacek wrote: post #83
solatic wrote: post #35

Earlier quoted context omitted.

> Every service --- and there are dozens... Most startups that I see trying to go with microservices too early do so while keeping a shared database between the services, so they're not really microservices, but a distributed monolith. This turns into a massive, massive pain. Doing microservices well means building out templates for CI/CD pipelines, templates for observability, figuring out how best to share (or not…

Sure, but I'm not thinking of small startups, but rather large ones like Doordash, Airbnb, Box, Square, Stripe; these places legitimately have lots of services, complex coupling patterns, and a need for some kind of indirection and dynamic placement to enable teams to get things done without coordinating every single PR. 37signals presumably doesn't have any of those problems. They should be using K8s. In the previou…

Nitpick: it may be helpful to not use the term "startups" when you mean companies like Doordash, Airbnb, Stripe etc. Those companies _were_ startups... a very long time ago. Same with 37signals.

This advice offered in the hope that it may improve communication!

Re: Kubernetes is hard

#109

Post by tptacek »

dgroshev wrote: post #101
tptacek wrote: post #83

Earlier quoted context omitted.

Sure, but I'm not thinking of small startups, but rather large ones like Doordash, Airbnb, Box, Square, Stripe; these places legitimately have lots of services, complex coupling patterns, and a need for some kind of indirection and dynamic placement to enable teams to get things done without coordinating every single PR. 37signals presumably doesn't have any of those problems. They should be using K8s. In the previou…

You are talking about a handful of unicorns, not just large startups, and even then only a subset of them — Shopify with their widely successful monolith comes to mind. By definition their requirements in their current unicorn state can't be the default. Yet we collectively behave as they are and everyone needs dynamic scheduling (and the rest of the kitchen sink).

Hey, we're on the same page about thinking critically about dynamic scheduling. :)

Re: Kubernetes is hard

#110

Post by sass_muffin »

honkycat wrote: post #47

I am consistently confused by all of the talk about how "hard" Kubernetes is. We spin up EKS. We install the newrelic and datadog log ingestion pods onto it, provided in a nice "helm" format. We install a few other resources via helm, like external secrets, and external dns, and a few others. Kubernetes EKS runs like a champ. My company saves 100k/mo by dynamically scaling our cloud services, all of which are running…

Same, we use EKS and a very similar setup, our workload has some pretty high throughput and scaling requirements. Works amazing for our team, wouldn't change it for anything else at this point. Very low maintenance effort since AWS manages the K8s infra.
Post reply on HN