Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nomad vs. Kubernetes

nomadproject.io

Post reply on HN
Print view View original

181–190 of 369 posts

Re: Nomad vs. Kubernetes

#181

Post by spookthesunset »

andrewstuart2 wrote: post #118
jeswin wrote: post #94

Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. The fundamental abstractions are as simple as they can be, representing concepts that you'd already be familiar with in a datacenter environment. A cluster has nodes (machines), and you can run multiple pods (which is the smallest deployable unit on the cluster) on each node. A pod ru…

From a system architecture perspective, kubernetes is very complex since it handles a multitude of complexities, but that's why it _can_ be very simple from a user's perspective. Most of the complexity can be ignored (until you want it). It's the same reason I still like to use postgres when I can versus NoSQL until I know I need the one feature I may not be able to achieve with postgres: automatic sharding for massi…

> From a system architecture perspective, kubernetes is very complex since it handles a multitude of complexities, but that's why it _can_ be very simple from a user's perspective.

I’ve been doing software for 20 years in one form or another. One of the things I’ve learned is the simpler and more polished something seems to the user, it is almost always because there is a hell of a lot of complexity under the covers to make it that way.

Making something that handles the 80% is easy. Every step closer to 100% becomes non-linear in a hurry. All that polish and ease of use took months of “fit & finish” timeboxes. It took tons of input from UX and product. It involved making very hard decisions so you, the user, don’t have to.

A good example is TurboTax online. People love to hate on their business practices (for good cause) but their UX handles like 98% of your common tax scenarios in an incredibly easy to use, highly polished way. Robinhood does a pretty good job too, in my opinion—there is a lot of polish in that app that abstracts away some pretty complex stuff.

Re: Nomad vs. Kubernetes

#182

Post by otabdeveloper4 »

farresito wrote: post #18
johnnycerberus wrote: post #9

Earlier quoted context omitted.

There you Go (pun intended). Go even replaced a low-level language like C++ and achieved the same result in the end. I don't know why I thought it's Java, probably the first Kube was initially in Java. It's even better that they managed to pull that off.

> Go even replaced a low-level language like C++ and achieved the same result in the end Did it, though? Honest question. I get the feeling that it ended up competing with Java (and Python) more than it ended up replacing C++. The C++ folks seem to be way more into Rust than Go.

Yes. The old-school PHP programmers seem to eventually move to Go. Going back to Go from C++ is almost unheard of.

Re: Nomad vs. Kubernetes

#183

Post by spookthesunset »

kbenson wrote: post #176
xtracto wrote: post #136

Earlier quoted context omitted.

> The rest of the features postgres (and friends) give easily (ACID etc) are very tricky to get right in a distributed system. But that's just basically a calculated tradeoff of Postgres (and several CP databases) trading Availability for Consistency.

Probably less calculated and more "that's what's available to offer stably right now that we can feasibly deliver, so that's what we'll do." Distributed RDBMS were not exactly cheap or common in open source a couple decades back. I don't think there was much of a choice to make.

I mean it is a trade off though. You cannot beat the speed of light. The further apart your database servers are, the more lag you get between them.

If you want a transactional, consistent datastore you are gonna have to put a lock on something while writes happen. And if you want consistency it means those locks need to be on all systems in the cluster. And the entire cluster needs to hold that lock until the transaction ends. If your DB’s are 100ms apart… that is a pretty large, non negotiable overhead on all transactions.

If you toss out being fully consistent as a requirement, things get much easier in replication-land. In that case you just fucking write locally and let that change propagate out. The complexity then becomes sorting out what happens when writes to the same record on different nodes conflict… but that is a solvable problem. There will be trade offs in the solution, but it isn’t going against the laws of physics.

Re: Nomad vs. Kubernetes

#184

Post by mkopinsky »

ledneb wrote: post #105
mkopinsky wrote: post #99

So far I've kept things simple, avoided k8s/HashiStack/etc by using docker compose with a simple docker-compose.yml for each server. This has been working well, but I'm starting to feel the pain points - HA requires copy-pasting yaml, I need to specify which services are on each server, and so on. What's the simplest next step I can take? I'd like something with (close to) the simplicity of docker compose, but ideall…

Assuming serverless is out if the question for your use case, have you tried spending a couple of days investigating a managed Kubernetes cluster with node autoscaling enabled? EKS, AKS, GKE... Honestly it sounds like you could be at the point where K8s is worthwhile.

I'm considering k8s, but that also means moving services from on-prem to AKS, getting INF to open up the necessary firewall rules to make the services reachable from on-prem, and so on. And as you said, it's definitely days of investigation. I'm not closed to the option.

Re: Nomad vs. Kubernetes

#185

Post by awinter-py »

wonder what the 'orchestration feature set' of 2 years in the future will be

speaking just for myself, feels like I want a system that runs a mix of serverless (like knative), normal containers, raw linux binaries (easy w/ fat binary systems like golang), language-specific packages like heroku does, maybe firecracker VMs.

hard to get knative right if you don't have a reverse proxy tightly integrated with your tool

Re: Nomad vs. Kubernetes

#186

Post by dilyevsky »

jeffbee wrote: post #149
tazjin wrote: post #2

As a Xoogler it's always seemed weird to me how Kubernetes was compared to Borg. Kubernetes covers a much larger set of things than Borg alone, and I don't necessarily think that's for the better. Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. Nowadays I don't have use-cases for either, but from playing around with Nomad it felt a lot more "Borg-y" th…

Borg is what k8s could be if there were any kind of ground rules and people were willing to call out some use cases as being legitimately stupid. Compared to k8s, Borg is a model of usability and simplicity.

Ground rules as in “you have to compile with 100s of these google3 libraries to run a simple app”?

Re: Nomad vs. Kubernetes

#187

Post by JelteF »

throwaway894345 wrote: post #108
q3k wrote: post #43

Earlier quoted context omitted.

> How do you deploy a thing to run on k8s? kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} One JSON/YAML file too uwieldy? Generate it using jsonnet/CUE/dhall/your favourite programming language. Or just talk directly to the Kubernetes API. You don't have to use Helm - in fact, you probably shouldn't be using Helm (as the whole idea of text templating YAML is... thoroughly ignorant in understanding what K…

> kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} This is fine for the first deploy, but if you delete a resource from your manifests then kubectl doesn’t try to delete it from the cluster, so in practice you need something like Terraform (or perhaps ArgoCD) which actually tracks state. And of course as you mention, you probably want to generate these manifests to DRY up your YAML so you can actually main…

I've been using the following command to great effect:

kubectl apply --kustomize manifests/ --prune --selector app=myapp

It cleans up old stuff from the cluster and also allows you to split your manifests across multiple files.

Re: Nomad vs. Kubernetes

#188

Post by SirMonkey »

ninkendo wrote: post #120
sheeshkebab wrote: post #57

Earlier quoted context omitted.

Maybe bc no one uses that in reality and use helm instead?

The first time I used helm was to set up JenkinsCI on a k8s cluster on AWS, and in the default configuration, it setup a public-internet ELB listener (with type=LoadBalancer) for Jenkins' internal JNLP port. Which pretty much means the public internet has root access to your jenkins cluster, by default. I had crypto miners using my k8s cluster within a couple of hours. That was also the last time I used helm.

You mean the helm-package you installed without reading the documentation? Hate the player, not the game.

Re: Nomad vs. Kubernetes

#189

Post by omneity »

awinter-py wrote: post #185

wonder what the 'orchestration feature set' of 2 years in the future will be speaking just for myself, feels like I want a system that runs a mix of serverless (like knative), normal containers, raw linux binaries (easy w/ fat binary systems like golang), language-specific packages like heroku does, maybe firecracker VMs. hard to get knative right if you don't have a reverse proxy tightly integrated with your tool

Wild guess, probably more than 2 years ahead (also kinda what I'm hoping for)

A natively distributed runtime providing similar capabilities to Docker + Nomad but running on "bare" WASM, providing a compilation target, distributed and clustering primitives right in your application code.

Imagine the nodejs cluster module, targeting nomad directly, or talking to an instance of a service running on another node, the communication and addressing would be handled by the runtime.

Similarly, cluster or instance level ports could be opened directly from your code and the distributed runtime will do the external/internal load balancing. Maybe taking it one step further and advertising services directly abstracting ports away.

Service mesh + Scheduling + Capability-based Runtime in one binary you could run on Linux. One can dream.

Re: Nomad vs. Kubernetes

#190

Post by p_l »

specialp wrote: post #177
andrewstuart2 wrote: post #118

Earlier quoted context omitted.

From a system architecture perspective, kubernetes is very complex since it handles a multitude of complexities, but that's why it _can_ be very simple from a user's perspective. Most of the complexity can be ignored (until you want it). It's the same reason I still like to use postgres when I can versus NoSQL until I know I need the one feature I may not be able to achieve with postgres: automatic sharding for massi…

It doesn't take long working with Nomad to hit the cases where you need to augment it. Now I know some of people enjoy being able to plug and play the various layers that you get in the complicated kitchen sink Kubernetes. We already had something that just ran containers and that was Mesos. They had the opinion that all the stuff like service discovery could be implemented and handled by other services like Marathon…

The important difference with k8s from my experience is that from the very early days it modeled a common IPC for any future interfaces, even if TPR/CRD tools some time to hash out. This means that any extra service can be simply added and then used with same general approach as all other resources.

This means you get to build upon what you already have, instead of doing everything from scratch again because your new infrastructure-layer service needs to integrate with 5 different APIs that have slightly different structure.

Post reply on HN