How we use HashiCorp Nomad • Hacker News Board

Re: How we use HashiCorp Nomad

#31

Post by kraig » Sat, Jun 06, 2020, 5:11 PM UTC

q3k wrote: post #26

speedgoose wrote: post #22

Earlier quoted context omitted.

Indeed. Helm offers great features but it suffers from the kubernetes unnecessary complexity and by using golang templates in YAML. When I started with kubernetes I converted my small Docker compose files to kubernetes files. Later I rewrote everything in helm charts. Now it's almost more YAML and golang templates lines than business logic lines in my applications. I'm considering to go back to Docker compose files.…

Highly recommend trying Jsonnet (via https://github.com/bitnami/kubecfg and https://github.com/bitnami-labs/kube-libsonnet ) as an alternative. It makes writing Kubernetes manifests much more expressive and integrates better with Git/VCS based workflows. Another language like Dhall or CUE might also work, but I'm not aware of a kubecfg equivalent for them. Jsonnet in general is a pretty damn good configuration langua…

Funny, I thought jsonnet was an even worse experience than templating yaml

Re: How we use HashiCorp Nomad

#32

Post by jlgaddis » Sat, Jun 06, 2020, 5:15 PM UTC

pier25 wrote: post #13

Off topic but... I've always wondered how Cloudflare can not charge for the bandwidth. Even the free plan is super generous (CDN, SSL, etc). How are they making a profit when AFAIK all other CDNs charge you for the bandwidth (and I assume they have to pay for to their providers)?

One reason is "settlement-free peering".

Re: How we use HashiCorp Nomad

#33

Post by pier25 » Sat, Jun 06, 2020, 5:17 PM UTC

nielsole wrote: post #14

pier25 wrote: post #13

Off topic but... I've always wondered how Cloudflare can not charge for the bandwidth. Even the free plan is super generous (CDN, SSL, etc). How are they making a profit when AFAIK all other CDNs charge you for the bandwidth (and I assume they have to pay for to their providers)?

See the second answer for a statement from cf https://webmasters.stackexchange.com/questions/88659/how-can...

I'm familiar with those terms but still.

There are people with free accounts moving GBs every month through their network and I imagine those free users must account for a very large percentage of their traffic.

Re: How we use HashiCorp Nomad

#34

Post by Pirate-of-SV » Sat, Jun 06, 2020, 5:18 PM UTC

rbjorklin wrote: post #29

waynesonfire wrote: post #20

Earlier quoted context omitted.

are you serious? you've got a major company touting a technology and a key component of it is broken?

I haven’t tested hard CPU quotas with Nomad but I suspect the issue mentioned above is due to cgroups/CFS and is also applicable to Kubernetes. See these slides for more details: https://www.slideshare.net/mobile/try_except_/optimizing-kub...

I recall that there's been some improvements to CFS to fix this since 2018.

Re: How we use HashiCorp Nomad

#35

Post by pier25 » Sat, Jun 06, 2020, 5:20 PM UTC

jlgaddis wrote: post #32

pier25 wrote: post #13

Off topic but... I've always wondered how Cloudflare can not charge for the bandwidth. Even the free plan is super generous (CDN, SSL, etc). How are they making a profit when AFAIK all other CDNs charge you for the bandwidth (and I assume they have to pay for to their providers)?

One reason is "settlement-free peering".

So what you're saying is that Cloudflare is so big they can reach these sort of deals with connectivity providers where they don't pay for bandwidth themselves?

Re: How we use HashiCorp Nomad

#36

Post by q3k » Sat, Jun 06, 2020, 5:21 PM UTC

kraig wrote: post #31

q3k wrote: post #26

Earlier quoted context omitted.

Highly recommend trying Jsonnet (via https://github.com/bitnami/kubecfg and https://github.com/bitnami-labs/kube-libsonnet ) as an alternative. It makes writing Kubernetes manifests much more expressive and integrates better with Git/VCS based workflows. Another language like Dhall or CUE might also work, but I'm not aware of a kubecfg equivalent for them. Jsonnet in general is a pretty damn good configuration langua…

Funny, I thought jsonnet was an even worse experience than templating yaml

My experience with jsonnet varied: there's good jsonnet code, and there's bad one, too. Just like with any programming language, you have to apply good software engineering practices. Text templated YAML, however, is terrible by design.

Re: How we use HashiCorp Nomad

#37

Post by monus » Sat, Jun 06, 2020, 5:21 PM UTC

4636760295 wrote: post #3

I worked at a hedge fund that also used nomad. The problem, however, is not how well it scales or whatever, but the fact that all the accompanying deployment info and literature is for kubernetes, and k8s has far more features. I like the quality of products from HashiCorp, but k8s is far, far, ahead of where nomad is. What I really want is better integration for Terraform and kubernetes. The current TF k8s support l…

Crossplane https://crossplane.io might be what you’re looking for with its bunch of controllers and a nice composition API.

One of the best features is that you can bundle a CR to request a MySQL database and it will be satisfied with whatever config is in your cluster so that app only declares the need but not care how it’s done.

Disclaimer: I’m one of the maintainers of Crossplane.

Re: How we use HashiCorp Nomad

#38

Post by [deleted] » Sat, Jun 06, 2020, 5:23 PM UTC

rayanimesh wrote: post #30

ognyankulev wrote: post #11

Earlier quoted context omitted.

Helm charts are declarative way of deploying app(s) and their accompanying resources.

> Helm charts are declarative way of deploying app(s) and their accompanying resources. How do you make helm chart deployment declarative? `helm install` is not declarative (in my understanding `kubectl apply` is declarative and `kubectl create` is not. Let me know if my understanding of declarative is wrong). Thanks.

[deleted]

Re: How we use HashiCorp Nomad

#39

Post by syllogism » Sat, Jun 06, 2020, 5:24 PM UTC

api wrote: post #4

4636760295 wrote: post #3

I worked at a hedge fund that also used nomad. The problem, however, is not how well it scales or whatever, but the fact that all the accompanying deployment info and literature is for kubernetes, and k8s has far more features. I like the quality of products from HashiCorp, but k8s is far, far, ahead of where nomad is. What I really want is better integration for Terraform and kubernetes. The current TF k8s support l…

Which features is Nomad missing? Feature count comparisons are meaningless unless the features are tied to actual important use cases. Lots of software is encrusted with rarely used features that just add complexity.

A few long-standing feature requests I've noticed a lot:

* No autoscaling

* Can't reserve entire CPU core: https://github.com/hashicorp/nomad/issues/289

* No way to run jobs sequentially: https://github.com/hashicorp/nomad/issues/419

Re: How we use HashiCorp Nomad

#40

Post by heipei » Sat, Jun 06, 2020, 5:27 PM UTC

Great writeup, even if I'm longing for more details about things like Unimog and their configuration management tool.

Pay close attention to the section where they described why they went with Nomad (Simple, Single-Binary, can schedule different workloads and not just docker, Integration with Consul). Nomad is so simple to run that you can run a cluster in dev mode with one command on your laptop (even MacOS native where you can try scheduling non-Docker workloads). I'd go so far as saying that it would even pay off to use Nomad as a job scheduler when you only have one machine and might have used systemd instead. You can wget the binary, write a small systemd unit for Nomad, then deploy any additional workloads with Nomad itself. By the time you have to scale to multiple machines you just add them to the cluster and don't have to rewrite your job definitions from systemd.