Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nomad vs. Kubernetes

nomadproject.io

Post reply on HN
Print view View original

211–220 of 369 posts

Re: Nomad vs. Kubernetes

#211

Post by pid-1 »

shrumm wrote: post #202

To most people, it's not Nomad vs Kubernetes - it's a choice between Nomad vs Managed Kubernetes. All major cloud providers offer a managed kubernetes service at minimal added cost to running the worker VMs yourself. With managed Kubernetes, the simplicity question is no longer obviously in Nomad's favour. As other comments allude to, Kubernetes as a user is pretty easy to master once you get used to its mental model…

Agreed. Managing a Nomad cluster alongside a Consul cluster does not require a PHD, but it's also not a walk in the park.

Hopefully Hashicorp will have managed Nomad soon.

Re: Nomad vs. Kubernetes

#212

Post by AzzieElbab »

gizdan wrote: post #191
wg0 wrote: post #179

Earlier quoted context omitted.

One of the things that I don't like about Nomad is HCL. It is a language that is mainly limited to HashiCorp tools and there's no wider adoption outside at least not to my knowledge. From the documentation: > Nomad HCL is parsed in the command line and sent to Nomad in JSON format via the HTTP API. So why not just JSON or even JSON at all and not MsgPack or just straight up HCL because that's over and over introduced…

JSON was designed for machine readability, HCL was designed for human readability. HCL requires a lot more code to parse and many more resources to keep in memory vs JSON. I think it completely makes sense to do it this way. K8s is the same. On the server it does everything in JSON. Your YAML gets converted prior to sending to K8s.

I don’t think Json was designed, it is just JavaScript objects plus Douglas Crockford spec. Having said that, HCL really doesn’t click with me

Re: Nomad vs. Kubernetes

#213

Post by ninkendo »

SirMonkey wrote: post #188
ninkendo wrote: post #120

Earlier quoted context omitted.

The first time I used helm was to set up JenkinsCI on a k8s cluster on AWS, and in the default configuration, it setup a public-internet ELB listener (with type=LoadBalancer) for Jenkins' internal JNLP port. Which pretty much means the public internet has root access to your jenkins cluster, by default. I had crypto miners using my k8s cluster within a couple of hours. That was also the last time I used helm.

You mean the helm-package you installed without reading the documentation? Hate the player, not the game.

No. No, no no.

This was legitimate a bug that they immediately fixed when I reported it... There is no legitimate reason to expose the master JNLP port to the internet, ever. The chart did not have a configuration option for this, it just exposed the JNLP port as part of the same k8s Service that exposed the web port. (The JNLP port is for jenkins workers to phone back home to the main instance, it's not for external use.)

"Just read the docs" is not an answer to a chart setup which is just plain broken.

Re: Nomad vs. Kubernetes

#214

Post by mac-chaffee »

shrumm wrote: post #202

To most people, it's not Nomad vs Kubernetes - it's a choice between Nomad vs Managed Kubernetes. All major cloud providers offer a managed kubernetes service at minimal added cost to running the worker VMs yourself. With managed Kubernetes, the simplicity question is no longer obviously in Nomad's favour. As other comments allude to, Kubernetes as a user is pretty easy to master once you get used to its mental model…

I'm currently trying to convince people that a managed k8s service is not that "simple", and that we can't "just spin up another cluster" without a great deal of operational overhead.

Some of the things that might still be needed in managed k8s instances: better ingress with ingress-nginx, cert-manager, monitoring/logging/alerting, tuning the alerts, integration with company SSO, security hardening.

If it's a multi-tenant cluster: LimitRanges/ResourceQuotas, NetworkPolicies, scripts to manage namespaces and roles, PodSecurityPolicies (or equivalent), onboarding offboarding procedures.

I'm sure you'd need similar things to have a proper production Nomad cluster too, so your point still stands. But at least for EKS/GKE clusters, they're pretty bare-bones.

Re: Nomad vs. Kubernetes

#215

Post by steveklabnik »

rkangel wrote: post #201
steveklabnik wrote: post #168

Earlier quoted context omitted.

Yes. This topic is long and complicated; maybe I'll write it up or give a talk or something someday. There is a lot of nuance.

I'd love to read a blog post on that. I completely see how it's a difficult line to walk - you don't want to be "Rust is better than all these rubbish languages" but you still want to provide people with information so that they can make informed choice.

Yup exactly. Comparisons are important, but you also really want to avoid building an “us vs them” culture.

Re: Nomad vs. Kubernetes

#216

Post by throwaway894345 »

JelteF wrote: post #187
throwaway894345 wrote: post #108

Earlier quoted context omitted.

> kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} This is fine for the first deploy, but if you delete a resource from your manifests then kubectl doesn’t try to delete it from the cluster, so in practice you need something like Terraform (or perhaps ArgoCD) which actually tracks state. And of course as you mention, you probably want to generate these manifests to DRY up your YAML so you can actually main…

I've been using the following command to great effect: kubectl apply --kustomize manifests/ --prune --selector app=myapp It cleans up old stuff from the cluster and also allows you to split your manifests across multiple files.

Whoa. I never knew about --prune.

Re: Nomad vs. Kubernetes

#217

Post by AlphaSite »

q3k wrote: post #43
jnsaff2 wrote: post #34

I liked Mesos when I worked on it, and it's been replaced by more modern tools like Nomad, but every time I have to work on k8s it's .. well it's being promoted like a cult, has a cult following and I think the whole thing is set up to suck up complexity and layering. How do you deploy a thing to run on k8s? One would think you deploy a manifest to it and that's it. Like yaml or json or hcl or whatever. No. The built…

> How do you deploy a thing to run on k8s? kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} One JSON/YAML file too uwieldy? Generate it using jsonnet/CUE/dhall/your favourite programming language. Or just talk directly to the Kubernetes API. You don't have to use Helm - in fact, you probably shouldn't be using Helm (as the whole idea of text templating YAML is... thoroughly ignorant in understanding what K…

I’m a big fan of kapp[1] for this sort of thing, it’s basically the same as kubectl apply, but with an app name and state tracking .

    kapp deploy -a my-app -f ./examples/simple-app-example/config-1.yml
It was created by folks at my employer but I use it because I just like it’s simplicity.

[1] https://carvel.dev/kapp/

Re: Nomad vs. Kubernetes

#218

Post by martijn9612 »

chucky_z wrote: post #205
tetha wrote: post #172

Earlier quoted context omitted.

Yes, I did not even consider that angle. And yes, nomad recommends 3 - 9 nodes in prod, maybe 11 or 13. The 3 are necessary to be able to tolerate one broken/maintained node and maintain the ability to schedule workloads. You can increase the number of tolerated nodes by increasing the number of nodes - 5 tolerate 2, 9 tolerate 3, 11 tolerate 5, 13 tolerate 6, but raft becomes slower with more nodes, because raft is…

I am currently running over 13000 nodes with a 5 server cluster. These are large boxes (90gb memory, 64 cores) but cpu usage is under 20% and memory usage is under 30%.

I'm sorry, what is the exact benefit of running multiple nodes on one piece of hardware? Just software failure resilience?

Re: Nomad vs. Kubernetes

#219

Post by throwaway894345 »

hosh wrote: post #174
throwaway894345 wrote: post #108

Earlier quoted context omitted.

> kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} This is fine for the first deploy, but if you delete a resource from your manifests then kubectl doesn’t try to delete it from the cluster, so in practice you need something like Terraform (or perhaps ArgoCD) which actually tracks state. And of course as you mention, you probably want to generate these manifests to DRY up your YAML so you can actually main…

Before there was Helm, I wrote my own tool and framework in Ruby (called Matsuri) to generate and manage manifests. I still use it. However, the source of truth is still on the kube api server. I did write in functionality to diff what is in the git repo and what is in the source of truth. There is an additional abstraction that bundles together resources so that a single command can converge them all together. The u…

> Before there was Helm, I wrote my own tool and framework in Ruby (called Matsuri) to generate and manage manifests.

I've built these kinds of things too in Python (or Starlark). It's an improvement over YAML, but I often feel the pain of dynamic typing.

> The Ruby code can access anything Ruby to generate the manifests, so I have a Turing-complete language and I can use class inheritance, mixins, method overrides.

I actually don't want any of these things in a dynamic configuration language. I kind of just want something that can evaluate expressions (including functions, objects, arrays, scalars, and pattern matching). If it's Turing complete that's fine but unnecessary. I explicitly don't want I/O (not in your list) or inheritance, mixins, method overrides, etc--unnecessary complexity.

Dhall probably comes the closest, but it has a tragic syntax and I'm not a good enough salesman to persuade real software teams to learn it. I've also heard good things about Cue, but my experiences playing with it have only been painful (steep learning curve, don't really see the upsides).

Re: Nomad vs. Kubernetes

#220

Post by rbonvall »

AzzieElbab wrote: post #212
gizdan wrote: post #191

Earlier quoted context omitted.

JSON was designed for machine readability, HCL was designed for human readability. HCL requires a lot more code to parse and many more resources to keep in memory vs JSON. I think it completely makes sense to do it this way. K8s is the same. On the server it does everything in JSON. Your YAML gets converted prior to sending to K8s.

I don’t think Json was designed, it is just JavaScript objects plus Douglas Crockford spec. Having said that, HCL really doesn’t click with me

Crockford himself says JSON was not invented but rather discovered.
Post reply on HN