Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nomad vs. Kubernetes

nomadproject.io

Post reply on HN
Print view View original

291–300 of 369 posts

Re: Nomad vs. Kubernetes

#291

Post by still_grokking »

cfigueir wrote: post #284
jeswin wrote: post #94

Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. The fundamental abstractions are as simple as they can be, representing concepts that you'd already be familiar with in a datacenter environment. A cluster has nodes (machines), and you can run multiple pods (which is the smallest deployable unit on the cluster) on each node. A pod ru…

The comparison is a bit misleading. As someone that has used both nomad and k8s at scale -- - Nomad is a scheduler. Clean and focused. It is very fast. I was an early user and encountered a number of bumps, but that's software. The people at Hashicorp are super sharp and lovely. - K8s is a lot more. It includes a scheduler, but in the simplest sense, it is a complete control-plane based on the control-loop pattern. Y…

I'm not sure it's a good idea to use Google's "monolithic cloud operation system".

Such a monoculture has the same issues as MS Windows had, even for the same reasons.

The "Unix Way" of simple tools interacting seems more reasonable. Especially when it comes to lock-in effects.

Re: Nomad vs. Kubernetes

#292

Post by still_grokking »

jnsaff2 wrote: post #34

I liked Mesos when I worked on it, and it's been replaced by more modern tools like Nomad, but every time I have to work on k8s it's .. well it's being promoted like a cult, has a cult following and I think the whole thing is set up to suck up complexity and layering. How do you deploy a thing to run on k8s? One would think you deploy a manifest to it and that's it. Like yaml or json or hcl or whatever. No. The built…

> but every time I have to work on k8s it's .. well it's being promoted like a cult, has a cult following

I would call this grass roots marketing. Some companies are very good at it. Maybe that ones have more know-how in the ad space than others?

Re: Nomad vs. Kubernetes

#293

Post by pronoiac »

mijoharas wrote: post #255
pronoiac wrote: post #158

Earlier quoted context omitted.

Consul is usually very reliable; when it breaks it can be very painful and mystifying. I've worked on removing it both from internal systems due to outages, and products, based on feedback.

Could you expand a little on your problems with Consul? (I have no experience with it myself).

It's been a while, so I'm fuzzy on some details:

* in a legacy system, a server, perhaps with the leader node, filled up disk space and became unhealthy, the consul agent kept reporting it and itself as healthy, and failover and gossip generally wedged

* in a dev environment, after we replaced some servers in the cluster, the other nodes noted cert changes and refused to work with the new servers

* second-hand, in self-hosted installations, it caused a number of hard-to-troubleshoot outages

* something about circular dependencies and going by "wait _n_ seconds" rather than by healthiness

It was reliable enough that it could gather a really significant blast radius, and it had different gnarly failure modes, so documentation could be irrelevant from case to case.

Re: Nomad vs. Kubernetes

#294

Post by still_grokking »

_wldu wrote: post #96
tekkk wrote: post #84

Earlier quoted context omitted.

What was wrong with ECS Fargate if you don't mind me asking? Too expensive? Too vendor locked-in?

We want to apply consistent policies and technical controls across multiple dev groups that have multiple AWS accounts. K8s seems to be a good solution for that.

I'm not a fan of AWS but to be fair it has all those features OOTB.

You can have AWS "organizations" managing multiple accounts. IAM will allow to set up any policies you want in your organization.

On the other hands side now you have a multitude of complexity added to your setup…

If I would be snarky I would see a case of "resume driven development". But let's just assume it was lack of time to find the simplest solution and it's than often preferred to just go with the flock as so many others can't be possibly wrong.

Re: Nomad vs. Kubernetes

#295

Post by still_grokking »

shrumm wrote: post #202

To most people, it's not Nomad vs Kubernetes - it's a choice between Nomad vs Managed Kubernetes. All major cloud providers offer a managed kubernetes service at minimal added cost to running the worker VMs yourself. With managed Kubernetes, the simplicity question is no longer obviously in Nomad's favour. As other comments allude to, Kubernetes as a user is pretty easy to master once you get used to its mental model…

> managed kubernetes service at minimal added cost

I dare to disagree. The costs are in fact horrific!

There are orders of magnitude in costs between running your stuff yourself vs the highest level of managed services on the cloud which is usually managed k8s.

(That also explains why there is so much marketing fuss, and push of management, to use k8s: It's by far the most profitable offering for the cloud providers).

Re: Nomad vs. Kubernetes

#296

Post by still_grokking »

pronoiac wrote: post #293
mijoharas wrote: post #255

Earlier quoted context omitted.

Could you expand a little on your problems with Consul? (I have no experience with it myself).

It's been a while, so I'm fuzzy on some details: * in a legacy system, a server, perhaps with the leader node, filled up disk space and became unhealthy, the consul agent kept reporting it and itself as healthy, and failover and gossip generally wedged * in a dev environment, after we replaced some servers in the cluster, the other nodes noted cert changes and refused to work with the new servers * second-hand, in se…

And what is the replacement for it?

Re: Nomad vs. Kubernetes

#297

Post by still_grokking »

papito wrote: post #104
gilbetron wrote: post #92

Earlier quoted context omitted.

> Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. How is Go not suited to those? I'm not seeing it - and are you comparing Go to Java or C++?

Go is a very hard language to work on as a team. It does not push a lot of conventions on you, and when it does, they just seem weird and anachronistic. Even the original k8s codebase was notoriously horrible in Go, and that was "from the source". Ironically, HashiCorp has a better Go codebase, and that's where I picked best practices from, not from Google. The problem with Go is that the Googlers fail to challenge s…

> As someone said, Go is the new C, skipping decades of new language design concepts.

Well, some are doomed to repeat their errors.

In fact C was even crappy for it's own time.

Re: Nomad vs. Kubernetes

#298

Post by GauntletWizard »

welkie wrote: post #289
throwaway894345 wrote: post #216

Earlier quoted context omitted.

Whoa. I never knew about --prune.

Yeah... I'm not really sure where this idea of not being able to use the tools available out of the box to deploy apps and do networking comes from. I deploy YAML. If I feel like my YAML is too big, I DRY it using kustomize. I can use --prune if I'm worried about stuff sticking around in the cluster. For networking, I... don't do anything? We get DNS built in. Just use the service name. What else is there to do?

Label selectors are hard. They might require knowledge as advanced as high-school geometry to understand. The ability to draw a Venn diagram isn't free, you know!

Re: Nomad vs. Kubernetes

#299

Post by still_grokking »

pdimitar wrote: post #40
hnlmorg wrote: post #24

Earlier quoted context omitted.

I've had just as many problems maintaining mature C++ projects as I have maintaining mature Go projects. Ultimately it all boils down to two things: - The projects being well written and well maintained from the outset - Personal preference I cannot overstate that second point. It really is about time developers stopped pushing their own personal preferences as if it's some kind of fact.

> It really is about time developers stopped pushing their own personal preferences as if it's some kind of fact. Agreed, so let's all go to Rust or OCaml because they have very strong static typing systems. ;) Both C++ and Go have plenty of warts, with C++ having way too many footguns and Go allowing you to use it as a dynamic language whenever you figure it's too much work to go through your problem with static typ…

> Agreed, so let's all go to Rust or OCaml because they have very strong static typing systems. ;)

Seems very reasonable. Especially OCaml should get more of the praise it deserves.

If you need to stay on the JVM there's Scala which allows (with some discipline) to write "when it compiles it works" code.

Re: Nomad vs. Kubernetes

#300

Post by still_grokking »

johnnycerberus wrote: post #23
tazjin wrote: post #16

Earlier quoted context omitted.

But it did not replace it. Google runs on Borg. Go lets you build things fast, but the lack of strong typing and the vast amount of language pitfalls make maintenance hard in the long run. The community also has the attitude to pretend that these pitfalls don't actually exist, which is very different from C++ where most peculiar behaviours are well-understood and controlled.

Sorry, I don't think I have been explicit in my comment. The language of K8s, a system which is the so-called direct descendant of Borg, replaced C++ at that layer. Also, Go is strongly typed.

Actually Google knew at the time they "designed" k8s that Borg doesn't scale due to fundamental design flaws in it's basic architecture. Still they reused the exact same architecture for k8s.

Who wants to know the details of those scaling issues can google for the Omega paper.

The "proper"™ solution to those design flaws was implemented in Mesos (and to my knowledge nowhere else until now).

Post reply on HN