Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we use HashiCorp Nomad

blog.cloudflare.com

Post reply on HN
Print view View original

141–150 of 166 posts

Re: How we use HashiCorp Nomad

#141

Post by ramaro »

q3k wrote: post #12
ognyankulev wrote: post #11

Earlier quoted context omitted.

Helm charts are declarative way of deploying app(s) and their accompanying resources.

Helm, however, is objectively terrible with its yaml-based templating language and zero practical modularity.

https://kapitan.dev/ is the one-stop shop that covers for true declarative configuration with either jsonnet, python (kadet) and jinja, amazing secret management with support of gkms, awskms, gpg, vault. It can also render helm charts!

It is simpler than other tools, because you can get started without even touching jsonnet or python or anything else, when using our generators. It does more than all the other tools combined, as it replaces helm+helmfile+gitcrypt or kustomize. It’s universal, so you can use it on non-kubernetes situations where other tools leave you high and dry.

With the new generator library, you can have 1 template and use it to configure many services. Check our examples at https://github.com/kapicorp/kapitan-reference

We have just released: * https://github.com/kapicorp/kapitan-reference a repo with examples for quick-start. It includes our generator as explained in https://medium.com/kapitan-blog/keep-your-ship-together-with...

* https://github.com/kapicorp/tesoro a secret “webhook controller” to seamlessly handle Kapitan secrets in your cluster. Better than sealed-secrets because there is no need to convert secrets and it supports KMS like google and aws together. Get started with our blog: https://medium.com/kapitan-blog or join our kubernetes slack on #kapitan

Re: How we use HashiCorp Nomad

#142

Post by orthoxerox »

q3k wrote: post #12
ognyankulev wrote: post #11

Earlier quoted context omitted.

Helm charts are declarative way of deploying app(s) and their accompanying resources.

Helm, however, is objectively terrible with its yaml-based templating language and zero practical modularity.

It is. But k8s has no convenient way of parameterizing releases that can beat Helm. A simple stateless application needs:

- a deployment - a service - an ingress - a config map (or several) - a secret (or several)

It's even worse for stateful applications.

And each of the resource definitions is 60% boilerplate, 35% application-specific and 5% release- or environment-specific.

Helm would probably be a nice and neat tool if it had stopped at maintaining a simple map of variable names to values. But since applications need things like "if the user said SQLite, add a pvc, a configmap and a secret and refer to them in the ss, if she said Postgres, go pull another chart, deploy it with these parameters, then add this configmap and this secret and refer to them in the ss", Helm is an overcomplicated mess.

Re: How we use HashiCorp Nomad

#143

Post by dmwilcox »

nine_k wrote: post #120
memexy wrote: post #112

Earlier quoted context omitted.

Pulumi has support both for Python and Go along with TypeScript and JavaScript. I never understand when people say operators don't like programming languages. Always seems weird to me that a person that constantly works with computers would prefer a static markup language instead of full programming language for managing the complexity of operating software in production environments. My original comment was about th…

A full programming language gives too much flexibility. In particular, this includes the flexibility allowing to shoot oneself in the foot. A person who constantly works with computers is painfully aware of the people's ability to screw things up by an honest mistake when programming them. The more complex the thing, the easier it is to make a mistake, because human attention is finite. Whenever you can limit the lan…

Having worked in several declarative config systems at a big tech company that tried to provide these guard rails. They errode!!

And it makes sense. As we demand more flexibility to not repeat ourselves (DRY) we get clever and add little features.

Suddenly you can stick an entire cluster config (or several!!) inside a lambda that returns the declared resources... And you've gone straight to hell. You're more than Turing complete by then and engineers are reinventing conditionals and loops with lambdas all over your config codebase.

Might as well have just started and left it at Python/etc.

Re: How we use HashiCorp Nomad

#144

Post by dmwilcox »

jFriedensreich wrote: post #45

One point i never get for companies operating their own hardware: If your problem was having a number of well known servers for the internal management services and you then move to a nomad cluster or kubernetes to schedule them dynamically, you end up with the same problem as before to schedule the well known nomad servers or kubernetes masters. So is the only advantage here that the nomad server images update less…

Not associated with CloudFlare but I've built similar stuff with Nomad.

The cost/maintenance trade-off works when you have more SPOF management hosts than Nomad servers (5). You decrease host images down to 2, Nomad server and client, versus N management images.

Though it does sound a bit like they're using config management rather than pre-build images.

Bonus, Nomad servers are more failure resistant using Raft consensus versus any N management hosts. And for discovery I found the optimal pattern is to put all of the Nomad servers in a "cluster" A record for clients to easily join (pattern works well for Consul too)

Re: How we use HashiCorp Nomad

#145

Post by dmwilcox »

aprdm wrote: post #105

Do people that aren't cloudflare scale really see the need for kubernetes and/or Nomad? Of the two Nomad seems much more sane because it does one thing only and is much simpler to manage and deploy. That said, having have used it, we are mostly moving away from it. Consul + Docker/Docker-compose with systemd in "a service per vm" model has proved much easier to administrate to our scale (couple of datacenters, ~1k VM…

Interesting, I went the other direction recently, from systemd to Nomad.

I was motivated by a move away from config management in favor of commands wrapping Nomad API calls. For a "devs on-call" model this was preferable to being gatekeepers of PRs against config management.

To glue the whole thing together I got Consul Connect going in the Nomad jobs, so service config complexity was comparable to docker-compose.

Saying this out loud makes me realize it was the organizational model I was pursuing that led me here (so called "production ownership"). And I'm not a big config management fan ;). I take it you have dedicated Ops teams or Devs willing to learn Ansible well?

Re: How we use HashiCorp Nomad

#146

Post by dmwilcox »

q3k wrote: post #10
api wrote: post #4

Earlier quoted context omitted.

Which features is Nomad missing? Feature count comparisons are meaningless unless the features are tied to actual important use cases. Lots of software is encrusted with rarely used features that just add complexity.

The free version of Nomad is missing, IIUC: - no quotas for teams/projects/organizations - no preemption (ie higher priority job preempts lower priority job) - no namespacing So generally it's somewhat useless in organizations where there are multiple different teams that should be able to coexist on a cluster without stepping on eachothers' toes, or even where you want a CI system to access the cluster in a safe man…

Enterprise version has all of those things ;)

For my personal use I don't need them and for a business Enterprise won't break the bank. You'd be surprised how much your management might be interested in having an escalation path if everything goes south and you need a hot fix. I can vouch that Enterprise support is worth it if you're on a small team that can't spend all day on this stuff

Re: How we use HashiCorp Nomad

#147

Post by toomuchtodo »

shaklee3 wrote: post #124
toomuchtodo wrote: post #94

Earlier quoted context omitted.

Question was rhetorical, I still appreciate the effort though.

Not at all. Hashicorp literally pays for all development of nomad. They're the only commits short of a small number of PRs. Kubernetes commits are from a wide array of companies, and Google is only one.

And just as with Kubernetes and Google, Nomad development can continue outside of Hashicorp if Hashicorp no longer decides to support it. Which org has a longer track record of deprecating almost everything they release? Not Hashicorp, and frankly, I’ll always trust Hashicorp versus Google based on the historical behavior and forward incentives of both.

Re: How we use HashiCorp Nomad

#148

Post by shaklee3 »

chucky_z wrote: post #131
shaklee3 wrote: post #127

Earlier quoted context omitted.

Nomad uses consul as the kv store, so it doesn't work without their other products.

Please make an attempt to understand a technology before making comments like this. Nomad has no requirement to store kv's, nor secrets. It's an entirely separate thing from Consul and Vault. It simply integrates with Consul, and Vault. It even runs its own raft store, so each product's backend is totally separate (Vault dropped it's reliance on other backends as of 1.4 and can run it's own raft store now). Nomad tem…

I understand nomad, and what you're saying is not how anyone runs it. Directly from their documentation:

"Nomad schedules workloads of various types across a cluster of generic hosts. Because of this, placement is not known in advance and you will need to use service discovery to connect tasks to other services deployed across your cluster. Nomad integrates with Consul to provide service discovery and monitoring."

So one of the very basic features of an orchestrator, which is service discovery, requires consul. Sure, I can use it without consul to just start jobs that don't communicate, but obviously that's that the normal use case of it, and you can see that by looking at their issues list.

Re: How we use HashiCorp Nomad

#149

Post by shaklee3 »

toomuchtodo wrote: post #147
shaklee3 wrote: post #124

Earlier quoted context omitted.

Not at all. Hashicorp literally pays for all development of nomad. They're the only commits short of a small number of PRs. Kubernetes commits are from a wide array of companies, and Google is only one.

And just as with Kubernetes and Google, Nomad development can continue outside of Hashicorp if Hashicorp no longer decides to support it. Which org has a longer track record of deprecating almost everything they release? Not Hashicorp, and frankly, I’ll always trust Hashicorp versus Google based on the historical behavior and forward incentives of both.

It's not quite the same. Hashicorp controls whether PRs get merged. Google does not control whether PRs get merged into kubernetes. There's a long list of companies that do, including IBM, redhat, Huawei, etc. Sure, you can fork it, but now you have a separate repo that requires people to know about it.

Re: How we use HashiCorp Nomad

#150

Post by toomuchtodo »

shaklee3 wrote: post #149
toomuchtodo wrote: post #147

Earlier quoted context omitted.

And just as with Kubernetes and Google, Nomad development can continue outside of Hashicorp if Hashicorp no longer decides to support it. Which org has a longer track record of deprecating almost everything they release? Not Hashicorp, and frankly, I’ll always trust Hashicorp versus Google based on the historical behavior and forward incentives of both.

It's not quite the same. Hashicorp controls whether PRs get merged. Google does not control whether PRs get merged into kubernetes. There's a long list of companies that do, including IBM, redhat, Huawei, etc. Sure, you can fork it, but now you have a separate repo that requires people to know about it.

I’m not trying to convince you. If k8s works for you, that’s great! I choose to work at places that pick Nomad instead. Best tool for the job.
Post reply on HN