Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nomad vs. Kubernetes

nomadproject.io

Post reply on HN
Print view View original

271–280 of 369 posts

Re: Nomad vs. Kubernetes

#271

Post by alexeldeib »

sofixa wrote: post #267
alexeldeib wrote: post #254

Earlier quoted context omitted.

https://www.vaultproject.io/docs/configuration/storage/raft It's new but I think is quickly becoming preferred. I found trying to setup nomad/consul/vault as described on the hashi docs creates some circular dependencies tbh (e.g. the steps to setup nomad reference a consul setup, the steps for vault mention nomad integration, but there's no clear path outside the dev server examples of getting there without reading…

It's not that circular - you start with Consul, add Vault and then Nomad, clustering them through Consul and configuring Nomad to use Vault and Consul for secrets and KV/SD respectively. And of course it can be done incrementally ( you can deploy Nomad without pointing it to Consul or Vault, and just adding that configuration later).

I don't mean a literal circular dependency. I mean the documentation doesn't clearly articulate how to get to having all 3x in a production ready configuration without bouncing around and piecing it together yourself.

For example, you mention starting with consul. But here's a doc on using Vault to bootstrap the Consul CA and server certificates: https://learn.hashicorp.com/tutorials/consul/vault-pki-consu...

So I need vault first. Which, oops, the recommended storage until recently for that was Consul. So you need to decide how you're going to bootstrap.

Vault's integrated Raft storage makes this a lot nicer, because you can start there and bootstrap Consul and Nomad after, and rely on Vault for production secret management, if you desire.

Re: Nomad vs. Kubernetes

#272

Post by zaat »

jeffnappi wrote: post #239
jeswin wrote: post #94

Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. The fundamental abstractions are as simple as they can be, representing concepts that you'd already be familiar with in a datacenter environment. A cluster has nodes (machines), and you can run multiple pods (which is the smallest deployable unit on the cluster) on each node. A pod ru…

> Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. Are you referring to actually spinning up and operating your own clusters here or utilizing managed k8s (e.g. GKE/EKS)? In my understanding, sure - using the managed services and deploying simple use cases on it might not be that big of a deal, but running and maintaining your own k…

Kubernetes the hard way is an educational resource, meant to server as a resource for those interested in deep diving into the platform, similar to Linux from scratch. Like the statement on the label:

> Kubernetes The Hard Way is optimized for learning, which means taking the long route to ensure you understand each task required to bootstrap a Kubernetes cluster.

> The results of this tutorial should not be viewed as production ready, and may receive limited support from the community, but don't let that stop you from learning!

If you want to spin up cluster that you actually want to use you'll pick one of the many available free or paid distros, and spinning up something like k3s with rancher, microk8s or even the pretty vanilla option of kubeadmin is pretty simple.

Re: Nomad vs. Kubernetes

#273

Post by llama052 »

zie wrote: post #249
llama052 wrote: post #242

Earlier quoted context omitted.

Honestly if you use a managed kubernetes provider it's pretty simple once you nail the concepts. Though you'll get hit with cloud provider issues every now and then but it's really not terrible. I'd manage services in Kubernetes before I start deploying VMs again, that's for sure.

Sure, you pay someone else to keep k8s alive, it's not so bad, but it's expensive to do that. You generally need a full-time team of people to keep a k8s deployment alive if you are running it yourself. I keep Nomad alive part-time. It's operationally simple and easy to wrap ones head around it and understand how it works.

A lot of the cloud providers don't bill directly for kubernetes management, instead it's just the node resources.

Either way, as another comment points out, Rancher and many other solutions make the orchestration of creating your own Kubernetes cluster really boring.

We run a few kubernetes clusters on premise, and for the longest time it was just 1 person running some Kubernetes clusters.. we even have other teams in QA/Engineering running their own with it.

Re: Nomad vs. Kubernetes

#274

Post by otterley »

alexeldeib wrote: post #252
otterley wrote: post #224

Earlier quoted context omitted.

Kubernetes' secrets management is nominal at best. It's basically just another data type that has K8S' standard ACL management around it. With K8S, the cluster admin has access to everything, including secrets objects. It's not encrypted at rest by default, and putting all the eggs in one basket (namely, etcd) means they're mixed in with all other control plane data. Most security practitioners believe secrets should…

It's "good enough" for most and extension points allow for filling the gaps. This also dodges the crux of GP's argument -- instead of running 1 cluster with 10 components, you now need a half dozen clusters with 1 component each, but oops they all need to talk to each other with all the same fun TLS/authn/authz setup as k8s components.

I'm a little confused. Why does the problem with K8S secrets necessitate having multiple clusters? One could take advantage of a more secure secrets system instead, such as Hashicorp Vault or AWS Secrets Manager.

Re: Nomad vs. Kubernetes

#275

Post by alexeldeib »

otterley wrote: post #274
alexeldeib wrote: post #252

Earlier quoted context omitted.

It's "good enough" for most and extension points allow for filling the gaps. This also dodges the crux of GP's argument -- instead of running 1 cluster with 10 components, you now need a half dozen clusters with 1 component each, but oops they all need to talk to each other with all the same fun TLS/authn/authz setup as k8s components.

I'm a little confused. Why does the problem with K8S secrets necessitate having multiple clusters? One could take advantage of a more secure secrets system instead, such as Hashicorp Vault or AWS Secrets Manager.

The point is that once you're talking about comparable setups, you need all of Vault/Nomad/Consul and the complexity of the setup is much more than just "one binary" as hashi likes to put it.

> So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did you gain really?

GP's point was already talking about running Vault clusters, not sure you realized we aren't only talking about nomad.

Re: Nomad vs. Kubernetes

#276

Post by otterley »

alexeldeib wrote: post #275
otterley wrote: post #274

Earlier quoted context omitted.

I'm a little confused. Why does the problem with K8S secrets necessitate having multiple clusters? One could take advantage of a more secure secrets system instead, such as Hashicorp Vault or AWS Secrets Manager.

The point is that once you're talking about comparable setups, you need all of Vault/Nomad/Consul and the complexity of the setup is much more than just "one binary" as hashi likes to put it. > So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did you gain really? GP's point was already talking about running Vault clusters, not sure you realized we aren't only talking about nom…

The only thing I was trying to say is that although K8S offers secrets "for free," it's not best practice to consider the control plane to be a secure secrets store.

Re: Nomad vs. Kubernetes

#277

Post by je42 »

zie wrote: post #249
llama052 wrote: post #242

Earlier quoted context omitted.

Honestly if you use a managed kubernetes provider it's pretty simple once you nail the concepts. Though you'll get hit with cloud provider issues every now and then but it's really not terrible. I'd manage services in Kubernetes before I start deploying VMs again, that's for sure.

Sure, you pay someone else to keep k8s alive, it's not so bad, but it's expensive to do that. You generally need a full-time team of people to keep a k8s deployment alive if you are running it yourself. I keep Nomad alive part-time. It's operationally simple and easy to wrap ones head around it and understand how it works.

nope. have a GKE cluster running "unattended" for months now. looks fine. ;)

Re: Nomad vs. Kubernetes

#278

Post by politelemon »

ookblah wrote: post #260

i would say the "simplest" setup for us has been ECS fargate (even thought there's some major lock in). Very easy to spin up a cluster and you don't have to manage the underlying structure. If you use docker-compose or equivalent for local dev you just define your tasks/services and launch. Even pulling in ENVs is easy with SSM.

What kind of lock-in? One of the reasons I started using it was due to the minimal lock-in, so we've arrived at different conclusions.

A task definition is required, which is sort of a docker-compose.yml analogue, but it's also possible to use docker compose directly with ECS Fargate. So my 'get out of AWS' is pretty simple, I already have the docker-compose.yml ready.

Re: Nomad vs. Kubernetes

#279

Post by mfer »

websap wrote: post #238
jeswin wrote: post #94

Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. The fundamental abstractions are as simple as they can be, representing concepts that you'd already be familiar with in a datacenter environment. A cluster has nodes (machines), and you can run multiple pods (which is the smallest deployable unit on the cluster) on each node. A pod ru…

I couldn't disagree with this more. I manage and run Kubernetes clusters as part of my job, and I can tell you that configuring, running and installing these clusters is no small feat. I don't know much about Nomad, but I would highly encourage most users to not think K8S is simple by any standard. Multiple cloud providers now provide ways to run your code directly, or your containers directly. Unless you have a reas…

> Despite its reputation, Kubernetes is actually quite easy to master for simple use cases.

This has so many assumptions rolled up in it. When we move from niche assumptions to the masses of app operators and cluster operators (two different roles) it tends to fall apart.

Which ultimately leads to things like...

> This has some great account of war stories - https://k8s.af/

We shouldn't assume other people are like us. It's worth understanding who folks are. Who is that ops person who wants to run Kubernetes at that insurance company or bank? Who is the person setting up apps to run for that auto company? What do they know, need, and expect?

When I answer these questions I find someone far different from myself and often someone who finds Kubernetes to be complicated.

Re: Nomad vs. Kubernetes

#280

Post by kerblang »

mijoharas wrote: post #255
pronoiac wrote: post #158

Earlier quoted context omitted.

Consul is usually very reliable; when it breaks it can be very painful and mystifying. I've worked on removing it both from internal systems due to outages, and products, based on feedback.

Could you expand a little on your problems with Consul? (I have no experience with it myself).

Not OP, but go look at the consul documentation. In fact, just look at "Configuration" page alone: https://www.consul.io/docs/agent/options - it goes FOREVER. Maybe you don't need most of that, but whatever you do need, you're going to have to find it somewhere in there.

And yes, the error messages can be utterly cryptic.

One thing: Don't ever let junior devs try to create a cluster out of their own laptops (yes they will think it sounds like a good idea) as this will be a never-ending nightmare.

Post reply on HN