Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nomad vs. Kubernetes

nomadproject.io

Post reply on HN
Print view View original

231–240 of 369 posts

Re: Nomad vs. Kubernetes

#231

Post by p_l »

darkwater wrote: post #132
q3k wrote: post #100

Earlier quoted context omitted.

> ... advocate for this. All what you are describing, which is basically what every hardcore k8s user/evangelist will tell you to do, it's reimplementing many, if not all, the features a Cloud provider is already giving you in their own resources. A well-designed KaaS offering from a cloud provider will do that by itself. GKE exposes GCP load balancers as an Ingress controller, IAM identities as Kubernetes RBAC ident…

> A well-designed KaaS offering from a cloud provider will do that by itself. GKE exposes GCP load balancers as an Ingress controller, IAM identities as Kubernetes RBAC identities, persistent disks as PVs, ... You just get them under a single declarative API. My experience with EKS on AWS tells me that it's not that simple, there are still many things to be glued together. I understand AWS historical position on K8s,…

With things like custom operators, especially crossplane (but also anything custom you cook up fast) or even custom operator wrapping AWS or GCP templates it's easy for me to offer curated verified solutions across all teams, instead of every one hacking off their own AWS/GCP/WTFcloud scripts to handle things. Even better than directly using cloud provider integration with ingress/service controllers, because I can provide specific limited variants of those APIs. And even without that, I can just use hooks system to blunt corners for the teams.

Re: Nomad vs. Kubernetes

#232

Post by glynnforrest »

Jedd wrote: post #147
glynnforrest wrote: post #61

> Flexible Workload Support This is Nomad's most underrated feature, IMHO. You don't have to use containers for everything if you don't want to. For example, if you're a golang shop you can run everything as native binaries and cut out docker completely. Nomad has much simpler networking, i.e. no web of iptables rules to figure out. You can add Consul connect as a service mesh if you need it, but if you don't, you ca…

This has been my biggest pain point - world+dog shares their helm charts, but blog posts or git*.com repos with non-trivial cluster recipes are rare. f.e. We're exploring cortexmetrics currently, and spinning the full stack version up on k8s (openshift) was straightforward. Porting all that spaghetti onto nomad would be a huge job, though part of the frustration is knowing someone, tucked away on a private network, h…

> the frustration is knowing someone, tucked away on a private network, has already done this.

Hard agree. I know this person and have been this person before.

I've toyed with the idea of writing a book of Nomad recipes and tips, I wonder if anyone would read it?

Also, watch this space, helm for Nomad may be coming soon: https://github.com/hashicorp/nomad-pack

Re: Nomad vs. Kubernetes

#233

Post by p_l »

mkopinsky wrote: post #184
ledneb wrote: post #105

Earlier quoted context omitted.

Assuming serverless is out if the question for your use case, have you tried spending a couple of days investigating a managed Kubernetes cluster with node autoscaling enabled? EKS, AKS, GKE... Honestly it sounds like you could be at the point where K8s is worthwhile.

I'm considering k8s, but that also means moving services from on-prem to AKS, getting INF to open up the necessary firewall rules to make the services reachable from on-prem, and so on. And as you said, it's definitely days of investigation. I'm not closed to the option.

You might also want to consider one of the simpler pre-packaged solutions for small k8s on premise clusters (like k3s)

Re: Nomad vs. Kubernetes

#234

Post by marvinblum »

Uberphallus wrote: post #124
marvinblum wrote: post #33

I've been running a production-grade Nomad cluster on Hetzner for the past 1 1/2 years and it's fantastic. It was amazingly easy to set up compared to Kubernetes (which I also did), the UI is awesome, updates haven't broken anything yet (as long as you follow the changelog) and it's stable. I really like the separation of concerns the HashiStack offers. You can start out just using Consul for your service meshing, an…

Hi, I see you mention the tiniest nodes in Hetzner there, whereas the Nomad documentation [0] talks about 3-5 server nodes in the 2-digit GiB memory range, which is what has kept me from trying Nomad as I find it insane. How much truth is there in the docs? [0] https://www.nomadproject.io/docs/install/production/requirem...

Sorry for the delayed response.

We started out with a three-node cluster using the smallest (1 vCPU/2GB RAM) VM you can get. Our initial requirement was to be able to do zero-downtime deployments and have a nice web UI to perform them (alongside the other advantages you get from a distributed system). We have now rescaled them to the next tier (2 vCPU/4GB RAM).

The hardware requirements depend on your workload. We process about 200 requests/sec right now and 600-700 per second on Saturdays (due to a larger client) and the nodes handle this load perfectly fine. All our services are written in Go and there is a single central Redis instance caching sessions.

Our database server is not part of the cluster and has 32 physical cores (64 threads) and 256GB RAM.

I say start out small and scale as you go and try to estimate the RAM usage of your workloads. The HashiStack itself basically runs on a calculator.

Re: Nomad vs. Kubernetes

#235

Post by sofixa »

gizdan wrote: post #197
throwaway923840 wrote: post #117

Earlier quoted context omitted.

They're both complex. But one of them has 10 times the components than the other, and requires you to use them. One of them is very difficult to install - so much so that there are a dozen different projects intended just to get it running. While the other is a single binary. And while one of them is built around containers (and all of the complexity that comes with interacting with them / between them), the other on…

> But one of them has 10 times the components than the other I've said this before. Kubernetes gives you a lot more too. For example in Nomad you don't have secrets management, so you need to set up Vault. Both Nomad and Vault need Consul for Enterprise set ups, of which Vault needs 2 Consul clusters for Enterprise setups. So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did y…

That's false. Vault has integrated storage and no longer needs Consul.

If you want to have the Enterprise versions( which aren't required), you just need 1 each of Nomad, Consul, Vault. Considering many people use Vault with Kubernetes anyway(due to the joke that is Kubernetes "secrets"), and Consul provides some very nice features and is quite popular itself, that's okay IMHO. Unix philosophy and all.

Re: Nomad vs. Kubernetes

#236

Post by marvinblum »

kalev wrote: post #52
marvinblum wrote: post #33

I've been running a production-grade Nomad cluster on Hetzner for the past 1 1/2 years and it's fantastic. It was amazingly easy to set up compared to Kubernetes (which I also did), the UI is awesome, updates haven't broken anything yet (as long as you follow the changelog) and it's stable. I really like the separation of concerns the HashiStack offers. You can start out just using Consul for your service meshing, an…

Great read, thank you.

I'm glad you like it :)

Re: Nomad vs. Kubernetes

#237

Post by rubenhak »

I don't agree with "Kubernetes is an orchestration system for containers originally designed by Google" statement. White it is not false, it creates a wrong impression of what K8s is.

Kubernetes is a cloud operating system which lets you run modern apps in any environment. One important component of course container orchestration, but it went far beyond just orchestrator. Kubernetes has a very powerful ecosystem, and it managed to unite almost all infrastructure vendors and projects. Its amazing to see how so many competing companies could agree on something. And that become K8s.

Nomad is great when you're working with VMs, but I don't see it is much relevant in the modern era of K8s and cloud-native.

Re: Nomad vs. Kubernetes

#238

Post by websap »

jeswin wrote: post #94

Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. The fundamental abstractions are as simple as they can be, representing concepts that you'd already be familiar with in a datacenter environment. A cluster has nodes (machines), and you can run multiple pods (which is the smallest deployable unit on the cluster) on each node. A pod ru…

I couldn't disagree with this more. I manage and run Kubernetes clusters as part of my job, and I can tell you that configuring, running and installing these clusters is no small feat. I don't know much about Nomad, but I would highly encourage most users to not think K8S is simple by any standard. Multiple cloud providers now provide ways to run your code directly, or your containers directly. Unless you have a reason to manage that complexity, save yourself some heartburn and make your developers happy that they just need to worry about running their code and not what happens when a seemingly innocuous change might cause a cascading failure.

This has some great account of war stories - https://k8s.af/

Re: Nomad vs. Kubernetes

#239

Post by jeffnappi »

jeswin wrote: post #94

Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. The fundamental abstractions are as simple as they can be, representing concepts that you'd already be familiar with in a datacenter environment. A cluster has nodes (machines), and you can run multiple pods (which is the smallest deployable unit on the cluster) on each node. A pod ru…

> Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones.

Are you referring to actually spinning up and operating your own clusters here or utilizing managed k8s (e.g. GKE/EKS)?

In my understanding, sure - using the managed services and deploying simple use cases on it might not be that big of a deal, but running and maintaining your own k8s cluster is likely far more of a challenge [1] than Nomad as I understand it.

[1] https://github.com/kelseyhightower/kubernetes-the-hard-way

Re: Nomad vs. Kubernetes

#240

Post by jbluepolarbear »

jeswin wrote: post #94

Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. The fundamental abstractions are as simple as they can be, representing concepts that you'd already be familiar with in a datacenter environment. A cluster has nodes (machines), and you can run multiple pods (which is the smallest deployable unit on the cluster) on each node. A pod ru…

It’s wouldn't have that reputation if it was easy. There’s too many things that can go wrong for it to be easy.
Post reply on HN