Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nomad vs. Kubernetes

nomadproject.io

Post reply on HN
Print view View original

251–260 of 369 posts

Re: Nomad vs. Kubernetes

#251

Post by kbenson »

lazide wrote: post #243
kbenson wrote: post #225

Earlier quoted context omitted.

Yes, but my point was there wasn't really a choice to make at that time, therefore no trade off. Even if I won $100 in the lotto today and had the money in hand, I wouldn't describe my choice which house I bought years ago as a calculated trade off between what I bought and some $10 million dollar mansion. That wasn't a feasible choice at that time. Neither was making a distributed RDBMS as an open source project dec…

Wasn’t MySQL (pre-Oracle) an open source distributed RDBMS decades ago? At least I remember running it using replication in early 2000’s

MySQL replication isn't really what I would consider a distributed RDBMS in the sense we're talking about, but it is in some senses. The main distinction being that you can't actually use it as a regular SQL interface. You have to have a primary/secondary and a secondary can't accept writes (if you did dual primary you had to be very careful about updates), etc. Mainly that you had to put rules and procedures in place for how it was used in your particular environment to allow for sharding or multiple masters, etc, because the underlying SQL system wasn't deterministic otherwise (also, the only replication available then was statement based replication, IIRC).

More closely matching would be MySQL's NDB clustered storage engine, which was released in late 2004.[1] Given that Postgres and MySQL both started ab out 1996, that's quite a time after initial release.

I spent a while in the early to mid 2000's researching and implementing dual master/slave or DRBD backed MySQL HA systems as a consultant, and the options available were very limited from what I remember. There's also probably a lot better tooling these days for developers to make use of separated read/write environments, whereas is seemed fairly limited back then.

1: https://en.wikipedia.org/wiki/MySQL_Cluster

Re: Nomad vs. Kubernetes

#252

Post by alexeldeib »

otterley wrote: post #224
gizdan wrote: post #197

Earlier quoted context omitted.

> But one of them has 10 times the components than the other I've said this before. Kubernetes gives you a lot more too. For example in Nomad you don't have secrets management, so you need to set up Vault. Both Nomad and Vault need Consul for Enterprise set ups, of which Vault needs 2 Consul clusters for Enterprise setups. So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did y…

Kubernetes' secrets management is nominal at best. It's basically just another data type that has K8S' standard ACL management around it. With K8S, the cluster admin has access to everything, including secrets objects. It's not encrypted at rest by default, and putting all the eggs in one basket (namely, etcd) means they're mixed in with all other control plane data. Most security practitioners believe secrets should…

It's "good enough" for most and extension points allow for filling the gaps.

This also dodges the crux of GP's argument -- instead of running 1 cluster with 10 components, you now need a half dozen clusters with 1 component each, but oops they all need to talk to each other with all the same fun TLS/authn/authz setup as k8s components.

Re: Nomad vs. Kubernetes

#253

Post by birdyrooster »

jnsaff2 wrote: post #34

I liked Mesos when I worked on it, and it's been replaced by more modern tools like Nomad, but every time I have to work on k8s it's .. well it's being promoted like a cult, has a cult following and I think the whole thing is set up to suck up complexity and layering. How do you deploy a thing to run on k8s? One would think you deploy a manifest to it and that's it. Like yaml or json or hcl or whatever. No. The built…

Because you have different team members managing different aspects of your software's deployment (if you have complex systems that you have moved into kubernetes). You can have a team for: authnz/rbac, networking, containerization/virtualization, service load balancing, app development, and secrets all while no one is stepping on each others toes deploying changes while using common language to talk about resources and automation.

Re: Nomad vs. Kubernetes

#254

Post by alexeldeib »

gizdan wrote: post #244
sofixa wrote: post #235

Earlier quoted context omitted.

That's false. Vault has integrated storage and no longer needs Consul. If you want to have the Enterprise versions( which aren't required), you just need 1 each of Nomad, Consul, Vault. Considering many people use Vault with Kubernetes anyway(due to the joke that is Kubernetes "secrets"), and Consul provides some very nice features and is quite popular itself, that's okay IMHO. Unix philosophy and all.

This is just false. I've run Vault in an Enterprise and unless something has changed in the last 12 months, Hashicorp's recommendation for Vaul has been 1 Consul cluster for Vault's data store, and 1 for it's (and other application's) service discovery. Sure Kubernetes's secrets is a joke by default, it's easily substituted by something that one actually considers a secret store.

https://www.vaultproject.io/docs/configuration/storage/raft

It's new but I think is quickly becoming preferred. I found trying to setup nomad/consul/vault as described on the hashi docs creates some circular dependencies tbh (e.g. the steps to setup nomad reference a consul setup, the steps for vault mention nomad integration, but there's no clear path outside the dev server examples of getting there without reading ALL the docs/references). There's little good docs in the way of bootstrapping everything 1 shot from scratch in the way most Kubernetes bootstrapping tools do.

Setting up an HA Vault/Consul/Nomad setup from scratch isn't crazy, but I'd say it's comparable level to bootstrapping k8s in many ways.

Re: Nomad vs. Kubernetes

#255

Post by mijoharas »

pronoiac wrote: post #158
proxysna wrote: post #14

Earlier quoted context omitted.

> I rarely hear anything about the Hashicorp alternative for service discovery and such though, so it would be interesting how that compares. Consul is used for service discovery. Fast, reliable and easily extensible. Yet to have a serious issue with it.

Consul is usually very reliable; when it breaks it can be very painful and mystifying. I've worked on removing it both from internal systems due to outages, and products, based on feedback.

Could you expand a little on your problems with Consul? (I have no experience with it myself).

Re: Nomad vs. Kubernetes

#256

Post by mijoharas »

johnnycerberus wrote: post #23
tazjin wrote: post #16

Earlier quoted context omitted.

But it did not replace it. Google runs on Borg. Go lets you build things fast, but the lack of strong typing and the vast amount of language pitfalls make maintenance hard in the long run. The community also has the attitude to pretend that these pitfalls don't actually exist, which is very different from C++ where most peculiar behaviours are well-understood and controlled.

Sorry, I don't think I have been explicit in my comment. The language of K8s, a system which is the so-called direct descendant of Borg, replaced C++ at that layer. Also, Go is strongly typed.

I'd argue that go is statically typed rather than strongly typed due to `interface{}`.

> There is no real agreement on what "strongly typed" means, although the most widely used definition in the professional literature is that in a "strongly typed" language, it is not possible for the programmer to work around the restrictions imposed by the type system. This term is almost always used to describe statically typed languages.

(random definition found googling).

Re: Nomad vs. Kubernetes

#257

Post by alexeldeib »

martijn9612 wrote: post #218
chucky_z wrote: post #205

Earlier quoted context omitted.

I am currently running over 13000 nodes with a 5 server cluster. These are large boxes (90gb memory, 64 cores) but cpu usage is under 20% and memory usage is under 30%.

I'm sorry, what is the exact benefit of running multiple nodes on one piece of hardware? Just software failure resilience?

I assume they meant a 5 server control plane supporting 13k worker nodes, not that they partitioned 5 large hosts into 13k smaller ones. It's a counterpoint to GP's "raft gets slow with more servers", I think.

Re: Nomad vs. Kubernetes

#258

Post by cpuguy83 »

papito wrote: post #104
gilbetron wrote: post #92

Earlier quoted context omitted.

> Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. How is Go not suited to those? I'm not seeing it - and are you comparing Go to Java or C++?

Go is a very hard language to work on as a team. It does not push a lot of conventions on you, and when it does, they just seem weird and anachronistic. Even the original k8s codebase was notoriously horrible in Go, and that was "from the source". Ironically, HashiCorp has a better Go codebase, and that's where I picked best practices from, not from Google. The problem with Go is that the Googlers fail to challenge s…

The original k8s codebase was Java, then rewritten in Go by Java developers. K8s is not a good example of Go, and it has nothing to do with the size of the project.

Re: Nomad vs. Kubernetes

#259

Post by k8sToGo »

zie wrote: post #249
llama052 wrote: post #242

Earlier quoted context omitted.

Honestly if you use a managed kubernetes provider it's pretty simple once you nail the concepts. Though you'll get hit with cloud provider issues every now and then but it's really not terrible. I'd manage services in Kubernetes before I start deploying VMs again, that's for sure.

Sure, you pay someone else to keep k8s alive, it's not so bad, but it's expensive to do that. You generally need a full-time team of people to keep a k8s deployment alive if you are running it yourself. I keep Nomad alive part-time. It's operationally simple and easy to wrap ones head around it and understand how it works.

You can also use something like Rancher or k3s to keep it alive part-time.

Re: Nomad vs. Kubernetes

#260

Post by ookblah »

i would say the "simplest" setup for us has been ECS fargate (even thought there's some major lock in). Very easy to spin up a cluster and you don't have to manage the underlying structure. If you use docker-compose or equivalent for local dev you just define your tasks/services and launch. Even pulling in ENVs is easy with SSM.
Post reply on HN