The funny thing is that blog post rant about mrsk which I thought was a Kubernetes implementation which is not. Docker ecosystem is just a mess.
Kubernetes is hard
81–90 of 164 posts
Re: Kubernetes is hard
#82> Kubernetes is complex and I think they are partially right Kubernetes is a distributed centralized operating system which itself depends on a distributed decentralized database, and has a varying network topology, permissions system, plugins, scheduler, storage, and much more, depending on how & where it was built, and runs applications as independent containerized environments (often deeply dependent on Linux kern…
> t allows dev teams to not worry about all these things; all they must do is to write a simple YAML file.
A "simple" YAML file.
Re: Kubernetes is hard
#8337signals is not like the typical large-scale startup. They have an extremely small team (around 30 people?), and just a couple of products. Large-scale startups use dynamic-scheduled cloud services in part to reduce coupling between teams. Every service --- and there are dozens --- is scheduled independently, and new teams can get spun up to roll out new services without too much intervention from other teams. When…
> Every service --- and there are dozens... Most startups that I see trying to go with microservices too early do so while keeping a shared database between the services, so they're not really microservices, but a distributed monolith. This turns into a massive, massive pain. Doing microservices well means building out templates for CI/CD pipelines, templates for observability, figuring out how best to share (or not…
Sure, but I'm not thinking of small startups, but rather large ones like Doordash, Airbnb, Box, Square, Stripe; these places legitimately have lots of services, complex coupling patterns, and a need for some kind of indirection and dynamic placement to enable teams to get things done without coordinating every single PR.
37signals presumably doesn't have any of those problems. They should be using K8s. In the previous thread, I suggested that I'd consider Nomad if I were them, but really, they don't need Nomad either. They need a yaml file mapping containers to hosts, and some tooling to do smooth deploys and to hook up a relatively static service discovery map. Which: that's what they built.
Re: Kubernetes is hard
#84Earlier quoted context omitted.
Or just app engine honestly. Works with docker containers so you can run the same simple stack locally as in prod. No need for more exotic serverless architectures. Generous free tier, too! Have only good things to say about it for quickly firing up a product.
> Does a product focused, non infra startup even need k8s? In my honest opinion people should be using Cloud Run. It’s by far Google’s best cloud product. > Or just app engine honestly. As a former App Engine PM and PM of Cloud Run, this warms my heart to hear--I'm glad folks love to use it :) It's been a few years since I've worked on these products, but they were designed to be composed and used by each other. Clou…
Oh I see, that's cool, thanks!
I had the idea that cloud run was like a (possibly auto-scaling?) abstraction of a container runtime or something so it makes sense that app engine uses it.
I don't think I have been aware of that fact though when using app engine in the past -- which is good, by the way, I don't want to know or care about this, just run my containers somehow :-)
Re: Kubernetes is hard
#85I am consistently confused by all of the talk about how "hard" Kubernetes is. We spin up EKS. We install the newrelic and datadog log ingestion pods onto it, provided in a nice "helm" format. We install a few other resources via helm, like external secrets, and external dns, and a few others. Kubernetes EKS runs like a champ. My company saves 100k/mo by dynamically scaling our cloud services, all of which are running…
Your company "saves" over 100k/month paying WAY too much for EKS, which is extremely expensive. If you're at any decent scale (looks like you are), then switch to GKE, or switch to on-prem and buy some hardware + a Kubernetes distro like Mirantis/Openshift/Tanzu. Heck, go run k3s on Hetzner and you won't have that much more work, but save literally millions at the scale you're talking about.
EKS is $0.10/hour, how is that "extremely expensive"? Or are you referring to an adjacent service?
Re: Kubernetes is hard
#86Kubernetes is hard because it's over-complicated and poorly designed. A lot of people don't want to hear that because it was created by The Almighty Google and people have made oodles of money being k8s gurus. After wasting two years chasing config files, constant deprecations, and a swamp of third-party dependencies that were supposedly "blessed" (all of which led to unnecessary downtime and stress), I swapped it al…
You can setup a solid k3s cluster in 30 minutes. I'm sorry you had a hard time but just because you didn't succeed at your attempt doesn't mean it actually is super hard.
Re: Kubernetes is hard
#87You need a decent sized team to run an on premise k8s infra. If you’re in the cloud use a managed k8s. It’s not for everyone in that I agree with the point the author makes. But if you have multiple teams doing app development k8s can be really nice. We do data movement and ML services. AKS has proven great for our use.
Your team needs scale with your demands. For many folk, a pretty boring setup is fine & requires no fiddling. Find a good gitops solution & many folks don't even need to think about kube, ever, ideally.
You do however need some folks who know Kube & systems. If you don't have them, hire some but not just for kube. If you can't do that, then yeah, it's a real personel investment.
Once you start having mtiple teams doing more complex stuff, you either need teams that work well together, or some kind of entity to figure out how not to make a mess. Turning on rbac is a significant early step towards being a mature, scale-out capable team.
I have yet to see an org scale past these needs myself, but most of me & my colleagues experience is at smaller/medium sized orgs.
Re: Kubernetes is hard
#8837signals is not like the typical large-scale startup. They have an extremely small team (around 30 people?), and just a couple of products. Large-scale startups use dynamic-scheduled cloud services in part to reduce coupling between teams. Every service --- and there are dozens --- is scheduled independently, and new teams can get spun up to roll out new services without too much intervention from other teams. When…
Who said anything about a large-scale startup? Kubernetes is approachable all the way down to N=1 employees.
I strongly disagree with your take on static vs dynamic scheduling. Static scheduling ties your hands early. In a mature organization, it is very much an optimization.
Dynamic scheduling forces a cattle-not-pets mentality out of the gate, which is great. It also gives you all the knobs to figure out HA, right-sizing and performance that you'd ever want, for whenever you're ready for them. It's considerably more laborious to rearrange or tune things with static scheduling. I've run the gamut here and Kubernetes is by far the easiest and most approachable way to manage a fleet from 1 to $BIG that I've ever encountered. If you want to treat it like a static scheduler, that is also trivial. It's not like there's some huge cost to doing so. It's basically a NOP.
37signals blew off their foot by doubling down on their sunk costs (read: capex) of metal. They clearly don't want to not think about KVM and F5s and SSH keys and all the other odds and ends that entirely solved away by managed services for reasonable prices.
Which is it? Are they too big for the cloud or too small? You can't have it both ways.
Re: Kubernetes is hard
#89> Kubernetes is complex and I think they are partially right Kubernetes is a distributed centralized operating system which itself depends on a distributed decentralized database, and has a varying network topology, permissions system, plugins, scheduler, storage, and much more, depending on how & where it was built, and runs applications as independent containerized environments (often deeply dependent on Linux kern…
> a simple YAML file
Oooooof
Re: Kubernetes is hard
#90Earlier quoted context omitted.
I disagree that production is hard in itself, I think the way people approach production makes it needlessly more difficult. It seems that when launching a service, everyone seems to try to get the entire stack in one shot, which is difficult. A much better approach is "1. Make it work 2. Make it good 3. Make it fast". Your initial prototypes need to define the core functionality, and then you incrementally build stu…
> 1. Make it work 2. Make it good 3. Make it fast Anecdotally, everytime someone tells me Kubernetes is overkill and then follows your approach. In a year they end up building the capabilities that come out of the box with Kubernetes, and of course they aren't as well thought out. Because they were done adhoc and as needed.
Why is that bad? If the business survived for long enough to have that problem, that's a win, not a failure. Being killed or at least hobbled by unnecessary complexity, on the other hand, is a thing in way more businesses than we like to admit.