How we use HashiCorp Nomad • Hacker News Board

Re: How we use HashiCorp Nomad

#21

Post by [deleted] » Sat, Jun 06, 2020, 4:44 PM UTC

chucky_z wrote: post #15

q3k wrote: post #10

Earlier quoted context omitted.

The free version of Nomad is missing, IIUC: - no quotas for teams/projects/organizations - no preemption (ie higher priority job preempts lower priority job) - no namespacing So generally it's somewhat useless in organizations where there are multiple different teams that should be able to coexist on a cluster without stepping on eachothers' toes, or even where you want a CI system to access the cluster in a safe man…

Preemption is coming in the next release in OSS :)

[deleted]

Re: How we use HashiCorp Nomad

#22

Post by speedgoose » Sat, Jun 06, 2020, 4:46 PM UTC

q3k wrote: post #12

ognyankulev wrote: post #11

Earlier quoted context omitted.

Helm charts are declarative way of deploying app(s) and their accompanying resources.

Helm, however, is objectively terrible with its yaml-based templating language and zero practical modularity.

Indeed. Helm offers great features but it suffers from the kubernetes unnecessary complexity and by using golang templates in YAML.

When I started with kubernetes I converted my small Docker compose files to kubernetes files. Later I rewrote everything in helm charts. Now it's almost more YAML and golang templates lines than business logic lines in my applications.

I'm considering to go back to Docker compose files. It's simple, readable, and easy to maintain.

Re: How we use HashiCorp Nomad

#23

Post by ianlevesque » Sat, Jun 06, 2020, 4:47 PM UTC

ognyankulev wrote: post #11

ianlevesque wrote: post #9

After trying out most of the kubernetes ecosystem in the pursuit of a declarative language to describe and provision services, Nomad was a breath of fresh air. It is so much easier to administer and the nomad job specs were exactly what I was looking for. I also noticed a lot of k8s apps encourage the use of Helm or even shell scripts to set up key pieces, which defeats the purpose if you are trying to be declarative…

Helm charts are declarative way of deploying app(s) and their accompanying resources.

Maybe I was doing it wrong but every guide was verb based - “helm install X”. My declarative ideal ended up being a text file full of helm install commands and that wasn’t what I wanted.

Re: How we use HashiCorp Nomad

#24

Post by phlakaton » Sat, Jun 06, 2020, 4:48 PM UTC

ognyankulev wrote: post #11

ianlevesque wrote: post #9

After trying out most of the kubernetes ecosystem in the pursuit of a declarative language to describe and provision services, Nomad was a breath of fresh air. It is so much easier to administer and the nomad job specs were exactly what I was looking for. I also noticed a lot of k8s apps encourage the use of Helm or even shell scripts to set up key pieces, which defeats the purpose if you are trying to be declarative…

Helm charts are declarative way of deploying app(s) and their accompanying resources.

Helm charts, round these parts, are a glorified yet byzantine templating system to spit out K8s manifests.

Re: How we use HashiCorp Nomad

#25

Post by candiddevmike » Sat, Jun 06, 2020, 4:48 PM UTC

The biggest hurdle with adopting nomad is the kubernetes ecosystem and related network effects. Things like operators only run on Kubernetes, and they're driving an entirely new paradigm of infrastructure and application management. HashiCorp is doing their best to keep up while supporting standard kube interfaces like CSI/CNI/CRI, but I don't know how they can possibly stay relevant with Kubernetes momentum.

In my opinion, HashiCorp should look at what Rancher did with K3s and offer something like that, integrated with the entire Hashi stack. The only reason most people choose nomad is the (initial) simplicity of it (which quickly goes away once you realize how "on an island" you are with solutions for ingress etc). Deliver kube with that simplicity and Integration and it's a much more compelling story than what Nomad delivers today.

Re: How we use HashiCorp Nomad

#26

Post by q3k » Sat, Jun 06, 2020, 4:48 PM UTC

speedgoose wrote: post #22

q3k wrote: post #12

Earlier quoted context omitted.

Helm, however, is objectively terrible with its yaml-based templating language and zero practical modularity.

Indeed. Helm offers great features but it suffers from the kubernetes unnecessary complexity and by using golang templates in YAML. When I started with kubernetes I converted my small Docker compose files to kubernetes files. Later I rewrote everything in helm charts. Now it's almost more YAML and golang templates lines than business logic lines in my applications. I'm considering to go back to Docker compose files.…

Highly recommend trying Jsonnet (via https://github.com/bitnami/kubecfg and https://github.com/bitnami-labs/kube-libsonnet) as an alternative. It makes writing Kubernetes manifests much more expressive and integrates better with Git/VCS based workflows. Another language like Dhall or CUE might also work, but I'm not aware of a kubecfg equivalent for them.

Jsonnet in general is a pretty damn good configuration language, not only for Kubernetes. And much more powerful than HCL.

Re: How we use HashiCorp Nomad

#27

Post by jeffbee » Sat, Jun 06, 2020, 4:54 PM UTC

"... here is the CPU usage over a day in one of our data centers where each time series represents one machine and the different colors represent different generations of hardware. Unimog keeps all machines processing traffic and at roughly the same CPU utilization."

Still a mystery to me why "balancing" has SO MUCH mindshare. This is almost certainly not the optimal strategy for user experience. It is going to be much better to drain traffic away from older machines while newer machines stay fully loaded, rather than running every machine at equal utilization factor.

Re: How we use HashiCorp Nomad

#28

Post by justincormack » Sat, Jun 06, 2020, 4:55 PM UTC

Nice that they did a kernel patch to fix the pivot root on initramfs problem, but sadly looking at replies it won't get merged https://lore.kernel.org/linux-fsdevel/20200305193511.28621-1...

Re: How we use HashiCorp Nomad

#29

Post by rbjorklin » Sat, Jun 06, 2020, 4:59 PM UTC

waynesonfire wrote: post #20

chucky_z wrote: post #5

Cloudflare, make sure you upgrade to 0.11.3, the new scheduling behavior is awesome for large clusters. Also a massive warning to anyone wanting to use hard cpu limits and cgroups (they do work in nomad it’s just not trivial), they don’t work like anyone expects and need to be heavily tested.

are you serious? you've got a major company touting a technology and a key component of it is broken?

I haven’t tested hard CPU quotas with Nomad but I suspect the issue mentioned above is due to cgroups/CFS and is also applicable to Kubernetes. See these slides for more details: https://www.slideshare.net/mobile/try_except_/optimizing-kub...

Re: How we use HashiCorp Nomad

#30

Post by rayanimesh » Sat, Jun 06, 2020, 5:08 PM UTC

ognyankulev wrote: post #11

ianlevesque wrote: post #9

After trying out most of the kubernetes ecosystem in the pursuit of a declarative language to describe and provision services, Nomad was a breath of fresh air. It is so much easier to administer and the nomad job specs were exactly what I was looking for. I also noticed a lot of k8s apps encourage the use of Helm or even shell scripts to set up key pieces, which defeats the purpose if you are trying to be declarative…

Helm charts are declarative way of deploying app(s) and their accompanying resources.

> Helm charts are declarative way of deploying app(s) and their accompanying resources.

How do you make helm chart deployment declarative? `helm install` is not declarative (in my understanding `kubectl apply` is declarative and `kubectl create` is not. Let me know if my understanding of declarative is wrong). Thanks.