361–370 of 681 posts
Earlier quoted context omitted.
Now set it up in 30 data centers around the world, with the ability for dozens of different teams to add and change their applications, and across multiple staging and QA environments.
Why do I need 30? 1 rack in a datacenter is plenty for a database backed web app with a million users.
In my case, I work for a CDN, so we need to have data centers all around the world.
My question is: Why is only k8s so popular when there are better alternatives for a large swath of users? I believe the answer is "Manufactured Hype". k8s is from a purely architectural standpoint the way to go, even for smaller setups, but the concrete project is still complex enough that it requires dozens of different setup tools and will keep hordes of consultants as well as many hosted solutions from Google/AWS/…
I got a bit disillusioned with k8s and looked at Nomad as an alternative. As a relatively noob sysadmin, I liked it a lot. Easy to deploy and easy to maintain. We've got a lot of mixed rented hardware + cloud VPS, and having one layer to unify them all seemed great. Unfortunately I had a hard convincing the org to give it a serious shot. At the crux of it, it wasn't clear what 'production ready' Nomad should look lik…
Consul by itself is the game-changer. Even in k8s it's a game-changer. It solves so many questions in an elegant way.
"How do I find and reach the things running in (orchestrator) with (unknown ip/random port) from (legacy)?" being the most important. You run 5 servers, and a relatively lightweight client on everything (which isn't even outright required, but it sure is useful!), and you get a _lot_ with that.
Consul provides multiple interfaces and ingress points to find everything. It also is super easy to operate, and has a pretty big community.
If you absolutely cannot have Consul, Nomad is still a really good batch job engine, and makes a very great "distributed cron," which is more extensible, scalable, and easy to use than something like Jenkins for the same task.
My team is pretty small (was 4 people, now 6) and we manage one of the worlds largest nomad and consul clusters (there are some truly staggeringly large users of Vault so I won't make that claim). Even when shit really hits the fan, everything is designed in a way that stuff mostly works; and there's enough operator friendly entry points that we can always figure out the problem.
I honestly couldn't tell you. What I can tell you, is that the unbelievable bloat in the complexity of our systems is going to bite us in the ass. I'll never forget when I joined a hip fintech company, and the director of eng told us in orientation that we should think of their cloud of services as a thousand points of light, out in space. I knew my days were numbered at exactly that moment. This company had 200k uni…
Does there exist a solution that does what kubernetes does but without the complexity? If such a tool does not exist do any of you feel that the creation of such a tool is within the realm of possibility? I would imagine all these knobs could have default configurations that 99% of all users would be okay with and that the knob should only be exposed in a small amount of cases.
My question is: Why is only k8s so popular when there are better alternatives for a large swath of users? I believe the answer is "Manufactured Hype". k8s is from a purely architectural standpoint the way to go, even for smaller setups, but the concrete project is still complex enough that it requires dozens of different setup tools and will keep hordes of consultants as well as many hosted solutions from Google/AWS/…
I got a bit disillusioned with k8s and looked at Nomad as an alternative. As a relatively noob sysadmin, I liked it a lot. Easy to deploy and easy to maintain. We've got a lot of mixed rented hardware + cloud VPS, and having one layer to unify them all seemed great. Unfortunately I had a hard convincing the org to give it a serious shot. At the crux of it, it wasn't clear what 'production ready' Nomad should look lik…
Edit: just noticed an actual Nomad user replied as well, and I like their answer better. Consider mine an addendum. :)
Batch workloads rarely require Consul, but for deploying your standard network services on Nomad: Consul is basically required. You could likely use any number of service mesh systems instead (either as sidecars, Docker network plugins, or soon CNI), but you'll be doing a lot of research and development on your own I'm afraid.
The Nomad team is by no means opposed to becoming more flexible in the future (and indeed better CNI support is landing soon as a first step), but we wanted to focus on getting one platform right and a pleasure to use before trying to genericize and modularize it.
My question is: Why is only k8s so popular when there are better alternatives for a large swath of users? I believe the answer is "Manufactured Hype". k8s is from a purely architectural standpoint the way to go, even for smaller setups, but the concrete project is still complex enough that it requires dozens of different setup tools and will keep hordes of consultants as well as many hosted solutions from Google/AWS/…
Nomad isn't really feature mature or user friendly enough, you still eventually need 100 bolt-ons. I think a Distributed OS is the only sane solution. Build the features we need into the kernel and stop futzing around with 15 abstractions to just run an isolated process on multiple hosts.
Linux (and the BSDs) are remarkably stable, festureful, and resilient operating systems. I would hate to give up such a strong foundation. Nomad can crash without affecting your running services. Nomad can be upgraded or reconfigured without affecting your running services. Nomad can be observed, developed, and debugged as a unit often without having to consider the abstractions that sit above or below it. The right number of abstractions is a beautiful thing. Just no more and no less. :)
I honestly couldn't tell you. What I can tell you, is that the unbelievable bloat in the complexity of our systems is going to bite us in the ass. I'll never forget when I joined a hip fintech company, and the director of eng told us in orientation that we should think of their cloud of services as a thousand points of light, out in space. I knew my days were numbered at exactly that moment. This company had 200k uni…
I honestly couldn't tell you. What I can tell you, is that the unbelievable bloat in the complexity of our systems is going to bite us in the ass. I'll never forget when I joined a hip fintech company, and the director of eng told us in orientation that we should think of their cloud of services as a thousand points of light, out in space. I knew my days were numbered at exactly that moment. This company had 200k uni…
I work at a fintech that runs a rails monolith on k8s and you would probably puke at how cheap our hosting cost is given our AUM. Engineering is a funny thing.
I’ll take a shot. k8s is popular because Docker solved a real problem and Compose didn’t move fast enough to solve orchestration problem. It’s a second order effect; the important thing is Docker’s popularity. Before Docker there were a lot of different solutions for software developers to package up their web applications to run on a server. Docker kind of solved that problem: ops teams could theoretically take anyt…
There are basically two relevant package managers. And say what you will about systemd, service units are easy to write.
It's weird to me that the tooling for building .deb packages and hosting them in a private Apt repository is so crusty and esoteric. Structurally these things "should" be trivial compared to docker registries, k8s, etc. but they aren't.
Earlier quoted context omitted.
Please don't criticize, condemn, or complain if you don't have anything constructive to add.
I'm not criticizing. I've actually used kubernetes and read the source code. It's a good tool, I just think its too much mental overhead for most companies since they won't use most of what it provides. If you're working on a large team with responsible parties who have a clearly defined roles it is a great tool but I've seen two person projects with startup infrastructure waste obscene amounts of time learning Kuber…