I have a stack which runs with a single docker-compose file, 13 services, nothings too fancy, tried to transform it to kubernettes (using kompose) and, my file was converted into almost 24 yaml files, Im not even lookings at those little gremnlins, and will stuck with simple docker-compose settings and docker swarm.
Kubernetes is hard
151–160 of 164 posts
Re: Kubernetes is hard
#152Earlier quoted context omitted.
This is a bad argument. You are basically admitting that companies need google-level resources to hire dozens (hundreds?) of infra engineers to maintain a K8s-type infra.
Where did I say that k8s needs that? I just said borg is more complex than k8s, so your argument doesn't even hold. Now, turning things around: companies need to hire more infra engineers, that's true.
> companies need to hire more infra engineers
Unless they don't.
Re: Kubernetes is hard
#153Earlier quoted context omitted.
> k8s permits me to evict workloads while obeying PDB — in previous orgs, "PDBs" (hell, we didn't even have a word to describe the concept) Odd, the parent makes it seem like resource budgets weren’t a thing before k8s.
I've never heard the term "resource budget" used to describe this concept before. Got a link? That'd be an odd set of words to describe it. To be clear, I'm not talking about budgeting RAM or CPU, or trying to determine do I have enough of those things. A PodDisruptionBudget describes the manner in which one is permitted to disrupt a workload: i.e., how can I take things offline? Your bog simple HTTP REST API service…
Sounds like a PDB isn’t a resource budget then. We were using that concept in ESX farms 20 years ago but it seems PDBs are more what more SREs would describe as minimum availability.
Re: Kubernetes is hard
#154I think setting up something similar without k8s is like, 100 times harder? I was never deeply into DevOps but a single short video and few doc pages told me how to bring up highly available, load-balanced 2-node cluster, and how to rollout new services and versions in minutes with zero downtime. I also can precisely control it, monitor all the logs and resources without leaving my working terminal for a minute. I would never be able to set-up an infra like this without kube in a timespan of one day with little prior DevOps knowledge. The complexity beast it tames into structure is just mind-blowing, and it's a virtue that it came out just being a bit "hard".
Re: Kubernetes is hard
#155Earlier quoted context omitted.
why on earth do you need a functional implementation of a consumed service for testing?
Iteration speed and blazing fast automated tests. When I discovered minio, I suddenly got much more confident coding against s3.
> Iteration speed and blazing fast automated tests.
Wholeheartedly agreed!
It is also nice to have that additional assurance of being able to self-host things (if ever necessary) and not being locked into a singular implementation. For example, that's why managed database offerings generally aren't that risky to use, given that they're built on already established projects (e.g. compatible with MySQL/MariaDB/PostgreSQL).
> When I discovered minio, I suddenly got much more confident coding against s3.
MinIO is pretty good, but licensing wise could become problematic if you don't work on something open source but ever want to run it in prod. Not really what this discussion is about, but AGPL is worth mentioning: https://github.com/minio/minio/blob/master/LICENSE
That said, thankfully S3 is so common that we have alternatives even to MinIO available, like Zenko https://www.zenko.io/ which is good for both local development as well as hosting in whatever environments necessary. I was actually about to mention Garage as well which seems better because it's a single executable but they also switched to AGPL, probably not an issue for local testing though: https://garagehq.deuxfleurs.fr/
Re: Kubernetes is hard
#156Earlier quoted context omitted.
I've never heard the term "resource budget" used to describe this concept before. Got a link? That'd be an odd set of words to describe it. To be clear, I'm not talking about budgeting RAM or CPU, or trying to determine do I have enough of those things. A PodDisruptionBudget describes the manner in which one is permitted to disrupt a workload: i.e., how can I take things offline? Your bog simple HTTP REST API service…
Sounds like a PDB isn’t a resource budget then. We were using that concept in ESX farms 20 years ago but it seems PDBs are more what more SREs would describe as minimum availability.
Maybe 'minimum instance availability' to be specific that were referring to instances of a service rather than an SLA.
Re: Kubernetes is hard
#157Kubernetes is hard because it's over-complicated and poorly designed. A lot of people don't want to hear that because it was created by The Almighty Google and people have made oodles of money being k8s gurus. After wasting two years chasing config files, constant deprecations, and a swamp of third-party dependencies that were supposedly "blessed" (all of which led to unnecessary downtime and stress), I swapped it al…
For a happy medium, check out Nomad. I've been managing our infrastructure on Nomad for years by myself, with upwards of 40 nodes (auto-scaled) and the number of problems we've had can be counted on one hand (and was almost always a simple user error or fixed by upgrading). I spend most of the time I would otherwise spend doing tedious ops shit actually building things. That said, Nomad and stateful services don't mi…
What do you do with stateful services?
Re: Kubernetes is hard
#158Earlier quoted context omitted.
For a happy medium, check out Nomad. I've been managing our infrastructure on Nomad for years by myself, with upwards of 40 nodes (auto-scaled) and the number of problems we've had can be counted on one hand (and was almost always a simple user error or fixed by upgrading). I spend most of the time I would otherwise spend doing tedious ops shit actually building things. That said, Nomad and stateful services don't mi…
What do you do with stateful services?
Run them on EC2 or whatever managed service whatever cloud host provides (RDS, S3, etc). It's possible to run stateful services on Nomad (and I assume k8s) but from my understanding the cost is extremely high, usually much higher than the benefit.
Re: Kubernetes is hard
#15937signals is not like the typical large-scale startup. They have an extremely small team (around 30 people?), and just a couple of products. Large-scale startups use dynamic-scheduled cloud services in part to reduce coupling between teams. Every service --- and there are dozens --- is scheduled independently, and new teams can get spun up to roll out new services without too much intervention from other teams. When…
I work for a company that routinely deploy very large scale software to airlines/airports/rail companies around the world. Millions of lines of mission critical server and mobile/browser/desktop client code.
We do it without the cloud, without micro-services, without Kubernetes etc. Just straight forward good old fashioned client/server monoliths. It’s simple. It works.
The reality is that 99% of people who think they need Kubernetes don’t actually need it. Almost all problems in software development are caused by the developers themselves. Not by the actual business problems.
Re: Kubernetes is hard
#160Earlier quoted context omitted.
> Every service --- and there are dozens... Most startups that I see trying to go with microservices too early do so while keeping a shared database between the services, so they're not really microservices, but a distributed monolith. This turns into a massive, massive pain. Doing microservices well means building out templates for CI/CD pipelines, templates for observability, figuring out how best to share (or not…
Sure, but I'm not thinking of small startups, but rather large ones like Doordash, Airbnb, Box, Square, Stripe; these places legitimately have lots of services, complex coupling patterns, and a need for some kind of indirection and dynamic placement to enable teams to get things done without coordinating every single PR. 37signals presumably doesn't have any of those problems. They should be using K8s. In the previou…
What on earth makes you think that the business domain of Airbnb/Doordash is any more complex than what 37signals is working on?
Almost all complexity in software is created by developers. 37signals seems to be one of the few companies that really understand this.