Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nomad vs. Kubernetes

nomadproject.io

Post reply on HN
Print view View original

241–250 of 369 posts

Re: Nomad vs. Kubernetes

#241

Post by maccard »

orthecreedence wrote: post #208
maccard wrote: post #85

Earlier quoted context omitted.

I can't run nomad on aws without managing a nomad cluster myself. I _can_ do that with k8s.

How have you found the managed k8s on AWS? I tend to shy away from their managed services until it has gone through a 4-5 year grace period, interested to know what your experience has been.

The short answer is it's less hassle than the local cluster I run locally for development purposes, to the point that were considering spinning up eks clusters for development use rather than running kind or k3s locally.

Re: Nomad vs. Kubernetes

#242

Post by llama052 »

websap wrote: post #238
jeswin wrote: post #94

Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. The fundamental abstractions are as simple as they can be, representing concepts that you'd already be familiar with in a datacenter environment. A cluster has nodes (machines), and you can run multiple pods (which is the smallest deployable unit on the cluster) on each node. A pod ru…

I couldn't disagree with this more. I manage and run Kubernetes clusters as part of my job, and I can tell you that configuring, running and installing these clusters is no small feat. I don't know much about Nomad, but I would highly encourage most users to not think K8S is simple by any standard. Multiple cloud providers now provide ways to run your code directly, or your containers directly. Unless you have a reas…

Honestly if you use a managed kubernetes provider it's pretty simple once you nail the concepts. Though you'll get hit with cloud provider issues every now and then but it's really not terrible.

I'd manage services in Kubernetes before I start deploying VMs again, that's for sure.

Re: Nomad vs. Kubernetes

#243

Post by lazide »

kbenson wrote: post #225
spookthesunset wrote: post #183

Earlier quoted context omitted.

I mean it is a trade off though. You cannot beat the speed of light. The further apart your database servers are, the more lag you get between them. If you want a transactional, consistent datastore you are gonna have to put a lock on something while writes happen. And if you want consistency it means those locks need to be on all systems in the cluster. And the entire cluster needs to hold that lock until the transa…

Yes, but my point was there wasn't really a choice to make at that time, therefore no trade off. Even if I won $100 in the lotto today and had the money in hand, I wouldn't describe my choice which house I bought years ago as a calculated trade off between what I bought and some $10 million dollar mansion. That wasn't a feasible choice at that time. Neither was making a distributed RDBMS as an open source project dec…

Wasn’t MySQL (pre-Oracle) an open source distributed RDBMS decades ago? At least I remember running it using replication in early 2000’s

Re: Nomad vs. Kubernetes

#244

Post by gizdan »

sofixa wrote: post #235
gizdan wrote: post #197

Earlier quoted context omitted.

> But one of them has 10 times the components than the other I've said this before. Kubernetes gives you a lot more too. For example in Nomad you don't have secrets management, so you need to set up Vault. Both Nomad and Vault need Consul for Enterprise set ups, of which Vault needs 2 Consul clusters for Enterprise setups. So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did y…

That's false. Vault has integrated storage and no longer needs Consul. If you want to have the Enterprise versions( which aren't required), you just need 1 each of Nomad, Consul, Vault. Considering many people use Vault with Kubernetes anyway(due to the joke that is Kubernetes "secrets"), and Consul provides some very nice features and is quite popular itself, that's okay IMHO. Unix philosophy and all.

This is just false. I've run Vault in an Enterprise and unless something has changed in the last 12 months, Hashicorp's recommendation for Vaul has been 1 Consul cluster for Vault's data store, and 1 for it's (and other application's) service discovery.

Sure Kubernetes's secrets is a joke by default, it's easily substituted by something that one actually considers a secret store.

Re: Nomad vs. Kubernetes

#246

Post by busterarm »

mac-chaffee wrote: post #214
shrumm wrote: post #202

To most people, it's not Nomad vs Kubernetes - it's a choice between Nomad vs Managed Kubernetes. All major cloud providers offer a managed kubernetes service at minimal added cost to running the worker VMs yourself. With managed Kubernetes, the simplicity question is no longer obviously in Nomad's favour. As other comments allude to, Kubernetes as a user is pretty easy to master once you get used to its mental model…

I'm currently trying to convince people that a managed k8s service is not that "simple", and that we can't "just spin up another cluster" without a great deal of operational overhead. Some of the things that might still be needed in managed k8s instances: better ingress with ingress-nginx, cert-manager, monitoring/logging/alerting, tuning the alerts, integration with company SSO, security hardening. If it's a multi-t…

As someone with 7-digit spend in GKE/EKS, I will agree with you that it is _anything but simple_.

Your developers aren't going to say that it's simple when Google force upgrades their cluster to a version that deprecates APIs in their yamls for a job they worked on 2 years ago and swiftly forgot about.

Then when you explain to them that Google insists on putting everyone on a force-upgrade treadmill, you can literally watch as the panic sets in on the faces of your engineering team managers/leads.

Nomad is a breeze in comparison to managed K8s.

Everyone that I've talked to that thinks Kubernetes is simple is barely using the thing and could likely save a lot of money and development effort using something like Nomad instead.

Re: Nomad vs. Kubernetes

#247

Post by toddnni »

Is it so that the fragmentation (the long list of distributions), or variety of configuration options, is a bad quality? This seems to be used as a counter argument to Kubernetes.

Probably, it could be end result of unconsistent design or bad technical choises. However, most likely it just means that there are multiple organizations and interest groups pushing changes and ideas to the project. This should be seen as a good thing. The downside is that there is no single source of best practices and this is confusing to newcomers. You just need to pick one distribution and trust the choises, or develop the competence and understanding.

And we could imagine that the userbase or the number of developers in single distribution (take OpenShift or Rancher) could be bigger that in Nomad itself.

Having said that, I still would like to see more stable Kubernetes landscape, and that has to happen eventually. The light distributions k3s and k0s are pushing things to nice direction.

OpenStack had similar, or maybe even worse, fragmentation and complexity issue when the hype was high. There were probably technically better alternatives (Eucalyptus?), but people (and companies) gathered around OpenStack and it won the round. However, comparing OpenStack to Kubernetes feels bad, as Kubernetes is technically far superior.

Re: Nomad vs. Kubernetes

#248

Post by jnsaff2 »

q3k wrote: post #169
LambdaComplex wrote: post #155

Earlier quoted context omitted.

> the whole idea of text templating YAML is... thoroughly ignorant in understanding what Kubernetes actually is Could you expand on that? It sounds like an interesting position (bordering on the philosophical), but I don't know enough about Kubernetes to gauge its accuracy for myself.

There's two things: 1) Text templating YAML is just bad. It's the serialized format of some structured data - instead of templating its text representation (and dealing with quoting, nindent, stringly-typed variables, and general YAML badness), just manipulate the structures directly before serializing them. For example, write some Python code that composes these structures in memory and then just serializes them to…

That is a very helpful summary. I generally favor declarative configuration (like CloudFormation) and I think it's mostly due to my work being infrastructure focused: "I need VPC, LB's, ASG with those immutable Images, a queue, buckets etc". But in my recent work with a customer where the infrastructure is "EKS with datastores, and Gitlab CI" .. most of the complexity is people creating abstractions on top of abstractions of stuff in helm (and also .gitlab-ci.yaml with tons of severy nested includes). And in this case the text templated yaml is really painful. Something that would be like CDK for k8s could actually be amazingly useful. Lots to ponder, thank you.

Re: Nomad vs. Kubernetes

#249

Post by zie »

llama052 wrote: post #242
websap wrote: post #238

Earlier quoted context omitted.

I couldn't disagree with this more. I manage and run Kubernetes clusters as part of my job, and I can tell you that configuring, running and installing these clusters is no small feat. I don't know much about Nomad, but I would highly encourage most users to not think K8S is simple by any standard. Multiple cloud providers now provide ways to run your code directly, or your containers directly. Unless you have a reas…

Honestly if you use a managed kubernetes provider it's pretty simple once you nail the concepts. Though you'll get hit with cloud provider issues every now and then but it's really not terrible. I'd manage services in Kubernetes before I start deploying VMs again, that's for sure.

Sure, you pay someone else to keep k8s alive, it's not so bad, but it's expensive to do that. You generally need a full-time team of people to keep a k8s deployment alive if you are running it yourself.

I keep Nomad alive part-time. It's operationally simple and easy to wrap ones head around it and understand how it works.

Re: Nomad vs. Kubernetes

#250

Post by paulgb »

LambdaComplex wrote: post #155
q3k wrote: post #43

Earlier quoted context omitted.

> How do you deploy a thing to run on k8s? kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} One JSON/YAML file too uwieldy? Generate it using jsonnet/CUE/dhall/your favourite programming language. Or just talk directly to the Kubernetes API. You don't have to use Helm - in fact, you probably shouldn't be using Helm (as the whole idea of text templating YAML is... thoroughly ignorant in understanding what K…

> the whole idea of text templating YAML is... thoroughly ignorant in understanding what Kubernetes actually is Could you expand on that? It sounds like an interesting position (bordering on the philosophical), but I don't know enough about Kubernetes to gauge its accuracy for myself.

> Could you expand on that? It sounds like an interesting position (bordering on the philosophical)

I'll make the philosophical case: text-level templating of a computer-readable format is almost always the wrong approach. It becomes unnecessarily hard for someone who later needs to read the code to ensure that it generates valid (both syntax and semantics) markup. It's also more hard for tooling to verify the validity, because the syntactic validity may depend on inputs to the program.

Compare the approaches of PHP and JSX. They both accomplish a similar goal (interleaved HTML and control flow), but JSX works at the element tree level and makes it impossible to generate poorly-formed HTML syntax -- it becomes a compile-time error (though it doesn't ensure semantics, e.g. a tbody outside of a table is legal). Compare with PHP, which very much allows you to generate invalid HTML, because it's just a text templating language.

(From what I can tell, Helm works more like PHP; if I'm wrong my philosophical position stands but might not apply here)

Post reply on HN