Any nice step by step tutorial how to get nomad vault and consul architecture running quickly? I’m especially interested in non container job.
Nomad vs. Kubernetes
321–330 of 369 posts
Re: Nomad vs. Kubernetes
#322Earlier quoted context omitted.
They're both complex. But one of them has 10 times the components than the other, and requires you to use them. One of them is very difficult to install - so much so that there are a dozen different projects intended just to get it running. While the other is a single binary. And while one of them is built around containers (and all of the complexity that comes with interacting with them / between them), the other on…
> But one of them has 10 times the components than the other I've said this before. Kubernetes gives you a lot more too. For example in Nomad you don't have secrets management, so you need to set up Vault. Both Nomad and Vault need Consul for Enterprise set ups, of which Vault needs 2 Consul clusters for Enterprise setups. So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did y…
You gained the suffering of dealing with split-brains in Consul and Vault ;-)
Re: Nomad vs. Kubernetes
#323Earlier quoted context omitted.
There's two things: 1) Text templating YAML is just bad. It's the serialized format of some structured data - instead of templating its text representation (and dealing with quoting, nindent, stringly-typed variables, and general YAML badness), just manipulate the structures directly before serializing them. For example, write some Python code that composes these structures in memory and then just serializes them to…
That is a very helpful summary. I generally favor declarative configuration (like CloudFormation) and I think it's mostly due to my work being infrastructure focused: "I need VPC, LB's, ASG with those immutable Images, a queue, buckets etc". But in my recent work with a customer where the infrastructure is "EKS with datastores, and Gitlab CI" .. most of the complexity is people creating abstractions on top of abstrac…
The tool that you want is probably Jsonnet. Take a look at Tanka or one of the other Jsonnet options. There is also Dhaal and a few others but Jsonnet has a lot more traction.
Re: Nomad vs. Kubernetes
#324Earlier quoted context omitted.
I'm currently trying to convince people that a managed k8s service is not that "simple", and that we can't "just spin up another cluster" without a great deal of operational overhead. Some of the things that might still be needed in managed k8s instances: better ingress with ingress-nginx, cert-manager, monitoring/logging/alerting, tuning the alerts, integration with company SSO, security hardening. If it's a multi-t…
As someone with 7-digit spend in GKE/EKS, I will agree with you that it is _anything but simple_. Your developers aren't going to say that it's simple when Google force upgrades their cluster to a version that deprecates APIs in their yamls for a job they worked on 2 years ago and swiftly forgot about. Then when you explain to them that Google insists on putting everyone on a force-upgrade treadmill, you can literall…
So instead of using higher level primitives that are widely used and tested and have many simple high quality integrations (external-dns, cert-manager, et al) you would recommend reinventing all of that on Nomad and then calling that "saving a lot of money and development effort"?
Yeah no thanks.
At this point k8s has "won" for all intents and purposes. It has gained critical mass, succeeding where other infrastructure management tools both open and closed source have failed.
Also API upgrades shouldn't be a problem, even if you were using beta APIs. They are only really troublesome if you decided to indulge yourself with some alpha APIs before they were fully baked. If you don't do that then you won't run into any problems.
Re: Nomad vs. Kubernetes
#325Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. The fundamental abstractions are as simple as they can be, representing concepts that you'd already be familiar with in a datacenter environment. A cluster has nodes (machines), and you can run multiple pods (which is the smallest deployable unit on the cluster) on each node. A pod ru…
Profoundly disagree. In fact, your statements are not true, since before you can run a pod you will most probably need to create a deployment. And then you'll need a service to expose your workload.
Anything but the most trivial workloads will also lead you into questions of how to mount volumes, cijfure) use configmaps and secrets, etc.
And that's not even touching the cluster configuration, which you can skip over if you are using a cloud provider that can provision it for you.
Re: Nomad vs. Kubernetes
#326To most people, it's not Nomad vs Kubernetes - it's a choice between Nomad vs Managed Kubernetes. All major cloud providers offer a managed kubernetes service at minimal added cost to running the worker VMs yourself. With managed Kubernetes, the simplicity question is no longer obviously in Nomad's favour. As other comments allude to, Kubernetes as a user is pretty easy to master once you get used to its mental model…
> managed kubernetes service at minimal added cost I dare to disagree. The costs are in fact horrific! There are orders of magnitude in costs between running your stuff yourself vs the highest level of managed services on the cloud which is usually managed k8s. (That also explains why there is so much marketing fuss, and push of management, to use k8s: It's by far the most profitable offering for the cloud providers)…
It's not. You generally only pay for the master clusters with no per-node costs over whatever instance type you are using and scheduling class (ondemand, reserved, spot, etc).
So not only are the masters relatively cheap, they cost is amortized over the cluster size.
AWS is probably very unhappy about this but they were forced into this by GKE pricing model. Google pushed people to adopt k8s not because they can make a lot of money from people using it but because they can take away a lot of money from AWS and increase workload portability, which benefits Google far more than AWS as they are in second place.
The most expensive managed services are things like RDS, MSK, Elasticache, etc. MSK in particular is pretty egregious with a 80% premium considering they haven't done any additional engineering like RDS.
Re: Nomad vs. Kubernetes
#327Earlier quoted context omitted.
Fun fact: Python and Go are both managed languages and need some kind of VM. I guess those people are just not educated enough. Those Ops people don't know much about programming in large usually. A lot of them for example think it's OK to write serous programs in Bash. This says it all, imho. It's OK when someone looking after admin stuff isn't a full blown programmer. It's a different kind of job after all. But thi…
Fun fact: There are formers devs among Ops, and sometimes vica versa. Go binaries are statically compiled usually, no VM or managed. It's a simple language for simple solutions, which is often underrated. Not that versed in Python, but there is CPython. Bash is totally ok in the hands of someone who uses it for good ;) Agree it has too quirky syntax when you need complexity, so not good for large stuff. Footguns are…
Go binaries still include the Go runtime which handles GCing etc., much like Python.
They're not VMs, but it's really a semantic difference, if you're comparing it to Java, especially now that GraalVM native compilation is gaining popularity.
Re: Nomad vs. Kubernetes
#328Earlier quoted context omitted.
The comparison is a bit misleading. As someone that has used both nomad and k8s at scale -- - Nomad is a scheduler. Clean and focused. It is very fast. I was an early user and encountered a number of bumps, but that's software. The people at Hashicorp are super sharp and lovely. - K8s is a lot more. It includes a scheduler, but in the simplest sense, it is a complete control-plane based on the control-loop pattern. Y…
I'm not sure it's a good idea to use Google's "monolithic cloud operation system". Such a monoculture has the same issues as MS Windows had, even for the same reasons. The "Unix Way" of simple tools interacting seems more reasonable. Especially when it comes to lock-in effects.
This is so far from the truth, I'm having a hard time imagining why you even think this. Kubernetes is practically the distributed embodiment of the Unix philosophy. You have a core set of interfaces and components that need to offer a particular API, and other than that, whether it is one program or many, written by one developer or hundreds, by a private company or via volunteers contributing to open source projects, is totally up to how you want to do it. You're free to use the original reference implementation that used to be owned by Google a decade ago before they open sourced it and donated it to the CNCF, but you certainly don't have to. Others have mentioned k3s, which is the busybox to the reference kubernetes GNU coreutils, all Kubernetes, plus ingress and network overlay, in a single binary, with am embedded sqlite db as the backing store instead of etcd. But k3s is still "Kubernetes." Kubernetes is a standard, much like POSIX. It's maybe unfortunate that the original reference implementation is also also named "kubernetes" because a lot of people seem to think that one is the only one you can use, and it has historically been complex to set up, but the reason for the complexity is it doesn't make any choices for you.
Imagine if you wanted to use a Unix operating system, but instead of choosing a Linux distro, you just read the POSIX standard and went out and found every required utility, plus a kernel, and had to figure out on your own how to get those to work together and create a system that can run application-level software. If you just go to kubernetes.io and follow the instructions on how to get up and running with the reference implementation, that is what you're doing. It makes no decisions at all for you. You can run external etcd, or use kubeadm to set it up for you. You can run it HA or on a single node. You can add whatever overlay network you want. You can use whatever container runtime engine you want. You can use whatever ingress controller you want, or none at all, and not have any external networking, just as you can install Linux From Scratch and not even bother to include networking if you want a disconnected system for some reason.
You have pretty much complete user freedom, and that is, in fact, the source and reason for a whole lot of complaints. Application developers and even most system administrators don't want to have to make that many decisions before they can even get to hello world. I believe Kelsey Hightower commented on this a while back, saying something to the effect that Kubernetes is not meant to be a developer platform. It's a framework for creating platforms.
Application developers, startups, and small business should almost never be using Kubernetes directly unless they're actually developing a platform product. Whether you use a "distro" like RKE2 or k3s or a managed service from a cloud provider, building out your own cluster using the reference kubernetes is the modern day equivalent of deploying a LAMP stack but doing it on top of Linux From Scratch.
Re: Nomad vs. Kubernetes
#329Earlier quoted context omitted.
They're both complex. But one of them has 10 times the components than the other, and requires you to use them. One of them is very difficult to install - so much so that there are a dozen different projects intended just to get it running. While the other is a single binary. And while one of them is built around containers (and all of the complexity that comes with interacting with them / between them), the other on…
> But one of them has 10 times the components than the other I've said this before. Kubernetes gives you a lot more too. For example in Nomad you don't have secrets management, so you need to set up Vault. Both Nomad and Vault need Consul for Enterprise set ups, of which Vault needs 2 Consul clusters for Enterprise setups. So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did y…
> Both Nomad and Vault need Consul for Enterprise set ups, of which Vault needs 2 Consul clusters for Enterprise setups. So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster.
This is incorrect. You don’t need consul for enterprise. Vault doesn’t need two consul clusters (it doesn’t need consul at all, if you don’t want it)
Re: Nomad vs. Kubernetes
#330Earlier quoted context omitted.
That's false. Vault has integrated storage and no longer needs Consul. If you want to have the Enterprise versions( which aren't required), you just need 1 each of Nomad, Consul, Vault. Considering many people use Vault with Kubernetes anyway(due to the joke that is Kubernetes "secrets"), and Consul provides some very nice features and is quite popular itself, that's okay IMHO. Unix philosophy and all.
This is just false. I've run Vault in an Enterprise and unless something has changed in the last 12 months, Hashicorp's recommendation for Vaul has been 1 Consul cluster for Vault's data store, and 1 for it's (and other application's) service discovery. Sure Kubernetes's secrets is a joke by default, it's easily substituted by something that one actually considers a secret store.
> This is just false.
No it isn’t.
> I've run Vault in an Enterprise
At this point I am starting to doubt that claim.