Live data from Hacker News

Nomad vs. Kubernetes

nomadproject.io

91–100 of 369 posts

Re: Nomad vs. Kubernetes

#91

I've run into so many issues with Nomad that it really doesn't make sense to compare the two. Many are well documented in Github issues and occur when utilizing other Hashicorp products.

I'm curious how responsive you have found them to issues you've identified.

Any particular problems you've experienced that have been long standing an unresolved?

I occasionally like to kick the tires on Nomad, but so far haven't found it compelling enough to switch away from my current solution, GKE (mostly because of how integrated it is with everything else I am doing and, at this point, familiarity).

So, curious to hear from others where the rough edges are.

Re: Nomad vs. Kubernetes

#92
post #2

As a Xoogler it's always seemed weird to me how Kubernetes was compared to Borg. Kubernetes covers a much larger set of things than Borg alone, and I don't necessarily think that's for the better. Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. Nowadays I don't have use-cases for either, but from playing around with Nomad it felt a lot more "Borg-y" th…

> Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either.

How is Go not suited to those? I'm not seeing it - and are you comparing Go to Java or C++?

Re: Nomad vs. Kubernetes

#93

> Flexible Workload Support This is Nomad's most underrated feature, IMHO. You don't have to use containers for everything if you don't want to. For example, if you're a golang shop you can run everything as native binaries and cut out docker completely. Nomad has much simpler networking, i.e. no web of iptables rules to figure out. You can add Consul connect as a service mesh if you need it, but if you don't, you ca…

> For example, if you're a golang shop you can run everything as native binaries and cut out docker completely.

Ironically you can also just deploy a go executable into an empty Docker container and it's basically the same as the raw executable, but with all the config abstracted to be the same as other containers'.

Re: Nomad vs. Kubernetes

#94
Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones.

The fundamental abstractions are as simple as they can be, representing concepts that you'd already be familiar with in a datacenter environment. A cluster has nodes (machines), and you can run multiple pods (which is the smallest deployable unit on the cluster) on each node. A pod runs various types of workloads such as web services, daemons, jobs and recurring jobs which are made available (to the cluster) as docker/container images. You can attach various types of storage to pods, front your services with load-balancers etc.

All of the nouns in the previous paragraph are available as building blocks in Kubernetes. You build your complex system declaratively from these simpler parts.

When I look at Nomad's documentation, I see complexity. It mixes these basic blocks with those from HashiCorp's product suite.

Re: Nomad vs. Kubernetes

#95

Been running Nomad for a while now at work and home, and it is such a fun project to work and tinker with. Great community, lots of plugins and other great stuff. After running k8s in prod and home Nomad felt like a breath of fresh air in all aspects.

What do you do with Nomad at home? What comprises your hardware? Just curious...

Re: Nomad vs. Kubernetes

#96
post #84
post #67

I have never deployed or built a K8s cluster, but recently I moved about a dozen AWS ECS Fargate workloads into a K8s cluster that a colleague of mine has setup. I was surprised. I really like it (from the perspective of a user/developer). I deploy my apps by updating a simple yaml file in a git repo. All my other practices (vim, go code with standard Makefiles, docker container image registry) are unchanged. I also…

What was wrong with ECS Fargate if you don't mind me asking? Too expensive? Too vendor locked-in?

We want to apply consistent policies and technical controls across multiple dev groups that have multiple AWS accounts. K8s seems to be a good solution for that.

Re: Nomad vs. Kubernetes

#97
post #84
post #67

I have never deployed or built a K8s cluster, but recently I moved about a dozen AWS ECS Fargate workloads into a K8s cluster that a colleague of mine has setup. I was surprised. I really like it (from the perspective of a user/developer). I deploy my apps by updating a simple yaml file in a git repo. All my other practices (vim, go code with standard Makefiles, docker container image registry) are unchanged. I also…

What was wrong with ECS Fargate if you don't mind me asking? Too expensive? Too vendor locked-in?

Not op, but I can't imagine ecs being cheaper than eks since ecs itself is free (you pay for fargate either way)

Ecs is pretty simple but does not have the large mindshare that kubernetes or nomad have.

Re: Nomad vs. Kubernetes

#98
post #87

Nomad is amazing. We've been using it alongside Consul for close to 2 years at this point at Monitoro.co[0]. We started by using it as a "systemd with a REST API" on a single server, and gradually evolved into multiple clusters with dozens of nodes each. It has been mostly a smooth ride, even if bumpy at moments, but operationally speaking Nomad is closer to Docker Swarm in simplicity and Kubernetes in terms of the f…

> Nomad is closer to Docker Swarm in simplicity and Kubernetes in terms of the feature set.

This a question I still need to google but what features does Kubernetes have that Docker Swarm needs?

Because the perceived complexity of Kubernetes just blows my mind, where Docker Swarm seems a lot more simpler for the same benefits, but my its just abstracted away?

I will say upfront im naive when it comes to container tech.

Re: Nomad vs. Kubernetes

#99
So far I've kept things simple, avoided k8s/HashiStack/etc by using docker compose with a simple docker-compose.yml for each server. This has been working well, but I'm starting to feel the pain points - HA requires copy-pasting yaml, I need to specify which services are on each server, and so on.

What's the simplest next step I can take? I'd like something with (close to) the simplicity of docker compose, but ideally being able to specify something like `instances: 2` and being able to route/load-balance to those instances from another service.

Re: Nomad vs. Kubernetes

#100
post #38

Earlier quoted context omitted.

> but all the clients our Kubernetes team works with all want separate clusters or testing, staging and preproduction. And I think that's one of the biggest issues with how people use Kubernetes these days (another candidate being insisting on drive-by deploying a cluster from scratch instead of deferring to cloud providers or a dedicated platform team that can plan for long-term maintenance). Kubernetes thrives in m…

> And I think that's one of the biggest issues with how people use Kubernetes these days (another candidate being insisting on drive-by deploying a cluster from scratch instead of deferring to cloud providers or a dedicated platform team that can plan for long-term maintenance). I don't really understand how you can say this and then... > Kubernetes thrives in multi-tenant environments: you get huge resource savings…

> ... advocate for this. All what you are describing, which is basically what every hardcore k8s user/evangelist will tell you to do, it's reimplementing many, if not all, the features a Cloud provider is already giving you in their own resources.

A well-designed KaaS offering from a cloud provider will do that by itself. GKE exposes GCP load balancers as an Ingress controller, IAM identities as Kubernetes RBAC identities, persistent disks as PVs, ... You just get them under a single declarative API.

> But you are taking the ownership and responsibility for this on your local platform/infra team.

With a platform team you're concentrating already existing responsibility into a team that can specialize in operational excellence - vs. that same responsibility being spread out across product teams that have to individually manage their own cloud resources, reinventing the wheel by writing the same terraform/{ansible,puppet,chef,...} boilerplate poorly. My experience is that these per-team bespoke AWS deployments are much more brittle than whatever a dedicated team can provide if given the responsibility and means to do things well.

> What if you screw something with CoreDNS? what if you break some RBAC roles used cluster-wide, while trying a change in the beta environment?

An outage is an outage, you roll back to stop the bleeding, investigate what happened and try to prevent whatever caused it from happening in the future. Neither of these examples are unsolvable in a multi-tenant environment, nor especially more likely to happen than similar screwups when using cloud provider resources.

Post reply on HN