[deleted]
Nomad vs. Kubernetes
71–80 of 369 posts
Re: Nomad vs. Kubernetes
#72Nomad seems much simpler to use and manage if you need to do simpler things, but Kubernetes allows you to do more. We use Kubernetes instead of Nomad at work but we are also using Consul in the Kubernetes cluster.
Nomad will still get you where you need to be. Aside from vm live migration i am yet to find a good example of a workload where nomad is straight up is unable to do what you need.
You mean you run VMs inside... Kubernetes? Am I misunderstanding something here?
Re: Nomad vs. Kubernetes
#73Nomad could use an official or semi-official distribution. Something that you could throw into any VM with minimal configuration and it would create a new or join an existing nomad cluster. I've been thinking about building such a thing on Arch (btw) but haven't acquired enough time-energy to do it.
Nomad + Consul does auto-clustering. All you have to do is specify in which mode your nomad binary is working and where your consul client is listening. For local development -dev mode exists for both Consul and Nomad.
Re: Nomad vs. Kubernetes
#74I liked Mesos when I worked on it, and it's been replaced by more modern tools like Nomad, but every time I have to work on k8s it's .. well it's being promoted like a cult, has a cult following and I think the whole thing is set up to suck up complexity and layering. How do you deploy a thing to run on k8s? One would think you deploy a manifest to it and that's it. Like yaml or json or hcl or whatever. No. The built…
You don't need any of that stuff. When I started with k8s, all I used were Yaml manifest files and kubectl apply. As I started using it more, i eventually moved up to using helm. I've been running production k8s for a few years now and haven't used helmsman or anything but helm yet.
At its core, Kubernetes is a controller manager. It's true that most Kubernetes systems include a lot of the Hashicorp equivalents but you could in theory remove the controllers and associated CRDs. Kubernetes has gradually moved to what I think is a relatively sensible set of defaults which you can get in different flavors (k0s, k3s, microkube, kubespray, etc.)
The comment about development and packaging tools can indeed present a problem. I tend to favor Helm for applications that need to be deployed more than once and Kustomize to differentiate between environments but I've definitely seen what I would consider horror stories. Even if you add a bit of GitOps, it's not too hard to understand. The horror stories seem to occur when teams try to map their previous workflows directly to k8s. Many times the worst k8s development workflows are created to enable teams that had broken workflows without k8s.
Re: Nomad vs. Kubernetes
#75Earlier quoted context omitted.
> Kubernetes also has better support for organization-wide multi-tenant clusters than Nomad seems to have That one's a little weird, I suppose you're right, but all the clients our Kubernetes team works with all want separate clusters or testing, staging and preproduction. They certainly don't want a multi-tenant cluster and share resources with other clients.
> but all the clients our Kubernetes team works with all want separate clusters or testing, staging and preproduction. And I think that's one of the biggest issues with how people use Kubernetes these days (another candidate being insisting on drive-by deploying a cluster from scratch instead of deferring to cloud providers or a dedicated platform team that can plan for long-term maintenance). Kubernetes thrives in m…
> And I think that's one of the biggest issues with how people use Kubernetes these days (another candidate being insisting on drive-by deploying a cluster from scratch instead of deferring to cloud providers or a dedicated platform team that can plan for long-term maintenance).
I don't really understand how you can say this and then...
> Kubernetes thrives in multi-tenant environments: you get huge resource savings and vastly simplified operations. Everyone in your organization gets access to all clusters, and they can just as easily deploy experimental best effort jobs or their development environment as they can deploy and/or inspect production jobs. Well set up quotas and priority classes mean that production jobs never run out of resources, while less important stuff (batch jobs, CI, someone's pet experiment) can continue to run on a best effort basis, just keeps getting preempted when production wants more resources.
... advocate for this. All what you are describing, which is basically what every hardcore k8s user/evangelist will tell you to do, it's reimplementing many, if not all, the features a Cloud provider is already giving you in their own resources. But you are taking the ownership and responsibility for this on your local platform/infra team. What if you screw something with CoreDNS? what if you break some RBAC roles used cluster-wide, while trying a change in the beta environment? I'm pretty sure there are (or will be) specific k8s tools to manage this but still, you are adding complexity and basically running another cloud provider inside a cloud provider for the sake of binpacking. For certain sizes of companies it might be worth the effort, but it is for sure not a silver bullet and probably applies to much less companies that many evangelists try to sell.
Re: Nomad vs. Kubernetes
#76Nomad could use an official or semi-official distribution. Something that you could throw into any VM with minimal configuration and it would create a new or join an existing nomad cluster. I've been thinking about building such a thing on Arch (btw) but haven't acquired enough time-energy to do it.
What you are seeking is called NixOS w/consul configured.
Re: Nomad vs. Kubernetes
#77I liked Mesos when I worked on it, and it's been replaced by more modern tools like Nomad, but every time I have to work on k8s it's .. well it's being promoted like a cult, has a cult following and I think the whole thing is set up to suck up complexity and layering. How do you deploy a thing to run on k8s? One would think you deploy a manifest to it and that's it. Like yaml or json or hcl or whatever. No. The built…
> How do you deploy a thing to run on k8s? kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} One JSON/YAML file too uwieldy? Generate it using jsonnet/CUE/dhall/your favourite programming language. Or just talk directly to the Kubernetes API. You don't have to use Helm - in fact, you probably shouldn't be using Helm (as the whole idea of text templating YAML is... thoroughly ignorant in understanding what K…
> kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml}
>
> One JSON/YAML file too uwieldy? Generate it using jsonnet/CUE/dhall/your favourite programming language. Or just talk directly to the Kubernetes API. You don't have to use Helm - in fact, you probably shouldn't be using Helm (as the whole idea of text templating YAML is... thoroughly ignorant in understanding what Kubernetes actually is).
Agreed 99%, but there is one useful thing that Helm provides over Kubectl+structured templating (Nix/dhall/whatever): pruning objects that are no longer defined in your manifest.
However, that's solvable without taking on all of Helm's complexity. For example, Thruster[0] (disclaimer: an old prototype of mine) provides a pruning variant of kubectl apply.
Re: Nomad vs. Kubernetes
#78Earlier quoted context omitted.
> How do you deploy a thing to run on k8s? kubectrl apply -f deploy.yaml should work, no? What forces you to use the sugar-coating?
Maybe bc no one uses that in reality and use helm instead?
That's a very odd and factually wrong generalisation.
I don't use helm at all, and I manage a large scale platform built on Kubernetes. Everything is either declared directly in YAMLs and deployed with `kubectl apply -f `, or rendered using Kustomize and, again, deployed using `kubectl apply -f -`.
Kustomize can be rough around the edges but it's predictable and I can easily render the YAMLs locally for troubleshooting and investigating.
Re: Nomad vs. Kubernetes
#79> Flexible Workload Support This is Nomad's most underrated feature, IMHO. You don't have to use containers for everything if you don't want to. For example, if you're a golang shop you can run everything as native binaries and cut out docker completely. Nomad has much simpler networking, i.e. no web of iptables rules to figure out. You can add Consul connect as a service mesh if you need it, but if you don't, you ca…
Flexible workload support doesn't sound great to me.
Good tools have very specific use-cases so as soon as you start talking about IIS on windows, VMs, containers etc. it just sounds like lots of the dependencies you are trying to remove in a scalable system.
Containers are relatively restrictive but they also enforce good discipline and isolation of problems which is a good idea imho. I would not want to continue to support our Windows servers running IIS, just too much to sort out.
Re: Nomad vs. Kubernetes
#80Looking forward to Nomad Packs, they where announced with their 1.2 beta[0]. I've really been missing something like k8s-at-home[1] for homelab Nomad setups. Don't know if they will become as versatile as Helm charts since Nomad is less of an abstraction than Kubernetes.
[0] https://www.hashicorp.com/blog/announcing-hashicorp-nomad-1-...