If someone's interested, i wrote a deeper dive into the matter a few months back, with concrete examples: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...
nicely covered.
Nomad vs. Kubernetes
141–150 of 369 posts
Re: Nomad vs. Kubernetes
#142Earlier quoted context omitted.
> How do you deploy a thing to run on k8s? kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} One JSON/YAML file too uwieldy? Generate it using jsonnet/CUE/dhall/your favourite programming language. Or just talk directly to the Kubernetes API. You don't have to use Helm - in fact, you probably shouldn't be using Helm (as the whole idea of text templating YAML is... thoroughly ignorant in understanding what K…
> kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} This is fine for the first deploy, but if you delete a resource from your manifests then kubectl doesn’t try to delete it from the cluster, so in practice you need something like Terraform (or perhaps ArgoCD) which actually tracks state. And of course as you mention, you probably want to generate these manifests to DRY up your YAML so you can actually main…
You can use -prune flag to apply to delete resources removed from your configs.
Re: Nomad vs. Kubernetes
#143Nomad seems much simpler to use and manage if you need to do simpler things, but Kubernetes allows you to do more. We use Kubernetes instead of Nomad at work but we are also using Consul in the Kubernetes cluster.
> Nomad seems much simpler to use and manage Agree, Nomad is so easy to get started and because of the simplicity of the architecture, very easy to maintain as well. > but Kubernetes allows you to do more ... We use Kubernetes instead of Nomad at work Same here, Kubernetes at work, Nomad for personal projects. But I have yet to find anything I cannot do in Nomad that you normally do in Kubernetes. Could you provide s…
The entire operator paradigm is kubernetes centric. You're missing out on all of the innovations around operations automation if you use nomad. Same with GitOps to an extent, HC pushes people to use terraform or waypoint for deployments while everyone else uses argo and tekton (or some other k8s runner).
Re: Nomad vs. Kubernetes
#144Earlier quoted context omitted.
> The homepage of Kubernetes seems to disagree with you. People have trouble understanding what k8s is, and what to use it for. That's fine, it'll take a while, but they will eventually understand what "Production-Grade Container Orchestration" really means when they start working with it.
Can you explain this? Having read it I have no idea what you think the differences are — it just sounds smug.
The first thing you start with on k8 is a Deployment. This will cover Container Scheduling, Replication, Orchestration on available Nodes, Secrets and volume bindings.
By just following this one tutorial (https://kubernetes.io/docs/tutorials/kubernetes-basics/deplo...), you already cover everything you used to do using docker-compose.
Now what ? You are going to learn about Services (https://kubernetes.io/docs/tutorials/kubernetes-basics/expos...), Ingresses (https://kubernetes.io/docs/concepts/services-networking/ingr...), Ingress-Controllers (https://kubernetes.io/docs/concepts/services-networking/ingr...), Persistent Volumes (https://kubernetes.io/docs/concepts/storage/persistent-volum...), Configuration, Pod Lifecycle, RBAC, Rolling Upgrades, Operator Pattern, ...
This is not about orchestrating containers anymore, it's a mix of network, configuration, storage APIs that just reunite everything you used to do with shell scripts under a fully declarative format. Then you realize, the _ACTUAL_ value of kubernetes isn't about the containers it can start-up, it's about being able to _MOVE_ those containers, their HTTP routing rules, their database, their backup schedule, their secrets, their configuration and everything else on totally different machines with different OS and different topology, just by running a kubectl apply.
Re: Nomad vs. Kubernetes
#145Earlier quoted context omitted.
Maybe bc no one uses that in reality and use helm instead?
Helm is useful if you need your software to run in many different places, and is widely known. This is why you see so many projects offering Helm charts; because you see them, they are set up to run in many environments. There is no reason to use it for your own software if you just have a single cluster.
Helms a pile of garbage but this isn’t really the fault of Helm. This is an issue with the chart or a failure to read the documentation of the chart.
People have got to stop just blindly running stuff off the internet.
Re: Nomad vs. Kubernetes
#146There seems to be plenty of reasons to run Nomad, compared to Kubernetes, but in what scenarios do Nomad lose out to Kubernetes? Is it simply a matter of Kubernetes being an open source project and Nomad being owned by HashiCorp?
We tried for multiple years to make Nomad work because it's simple. We're already enterprise customers of Hashicorp, too. We love Hashicorp! Nomad is a great scheduler and the UI is wonderful, but there is no story around network security. Example: You have postgres running on dedicated metal, and you want to restrict which nomad services have access to postgres. Consul Connect allows you to enforce access policies, but these work more like haproxy with automated service discovery. There is no batteries-included way to prevent traffic from leaving your container (or identify which container the traffic is originating from). You can use a custom CNI plugin to give each container an IP address from a unique subnet per service (and restrict traffic at your firewall/ selectively add subnets to iptabels on your postgres server), but now we're adding bespoke complexity. We brought this problem up to the Hashicorp internal teams over many calls, but ultimately they said we'd need to pay for a 3rd party overseas consultant to advise us on implementation. They weren't sure what to do. K8s is certainly more complex out the gate, but you don't feel like you're the first person in the world to implement.
That said, I think Nomad is a few years away from being something truly amazing. If they can solve the network security problem (beyond optimistic documentation), I think it'll be amazing. For now, it's just a good scheduler.
Re: Nomad vs. Kubernetes
#147> Flexible Workload Support This is Nomad's most underrated feature, IMHO. You don't have to use containers for everything if you don't want to. For example, if you're a golang shop you can run everything as native binaries and cut out docker completely. Nomad has much simpler networking, i.e. no web of iptables rules to figure out. You can add Consul connect as a service mesh if you need it, but if you don't, you ca…
This has been my biggest pain point - world+dog shares their helm charts, but blog posts or git*.com repos with non-trivial cluster recipes are rare.
f.e. We're exploring cortexmetrics currently, and spinning the full stack version up on k8s (openshift) was straightforward. Porting all that spaghetti onto nomad would be a huge job, though part of the frustration is knowing someone, tucked away on a private network, has already done this.
Re: Nomad vs. Kubernetes
#148Earlier quoted context omitted.
You mean you run VMs inside... Kubernetes? Am I misunderstanding something here?
Not only k8s, but i ran VM's with both Nomad and K8s. Nomad supports VM workloads out of the box. K8s requires Kubevirt [0]. [0] https://www.nomadproject.io/docs/drivers/qemu [1] https://kubevirt.io/
I knew about nomad (and it makes sense given it has from the start a more corporate, on-prem audience), didn't know/remember about KubeVirt. Hopefully I won't have any use for it in my scenario but thanks for the link!
Re: Nomad vs. Kubernetes
#149As a Xoogler it's always seemed weird to me how Kubernetes was compared to Borg. Kubernetes covers a much larger set of things than Borg alone, and I don't necessarily think that's for the better. Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. Nowadays I don't have use-cases for either, but from playing around with Nomad it felt a lot more "Borg-y" th…
Borg is what k8s could be if there were any kind of ground rules and people were willing to call out some use cases as being legitimately stupid. Compared to k8s, Borg is a model of usability and simplicity.
Re: Nomad vs. Kubernetes
#150I've been running a production-grade Nomad cluster on Hetzner for the past 1 1/2 years and it's fantastic. It was amazingly easy to set up compared to Kubernetes (which I also did), the UI is awesome, updates haven't broken anything yet (as long as you follow the changelog) and it's stable. I really like the separation of concerns the HashiStack offers. You can start out just using Consul for your service meshing, an…
Hi, I see you mention the tiniest nodes in Hetzner there, whereas the Nomad documentation [0] talks about 3-5 server nodes in the 2-digit GiB memory range, which is what has kept me from trying Nomad as I find it insane. How much truth is there in the docs? [0] https://www.nomadproject.io/docs/install/production/requirem...
This very much depends on your workload, number of jobs and complexity of scheduling. Our Nomad servers have 4GB of memory in the VM and are using about 0.5 - 1G at a low three-digit number of jobs.
Hashicorp is doing a smart but normal thing for on-prem sizing there - they are recommending specs which ensure you have no problems for a lot of workload sizes. And "workload" can grow very large there, since a single DC can handle up to 5k clients and such.