Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nomad vs. Kubernetes

nomadproject.io

Post reply on HN
Print view View original

341–350 of 369 posts

Re: Nomad vs. Kubernetes

#341

Post by ksec »

still_grokking wrote: post #300
johnnycerberus wrote: post #23

Earlier quoted context omitted.

Sorry, I don't think I have been explicit in my comment. The language of K8s, a system which is the so-called direct descendant of Borg, replaced C++ at that layer. Also, Go is strongly typed.

Actually Google knew at the time they "designed" k8s that Borg doesn't scale due to fundamental design flaws in it's basic architecture. Still they reused the exact same architecture for k8s. Who wants to know the details of those scaling issues can google for the Omega paper. The "proper"™ solution to those design flaws was implemented in Mesos (and to my knowledge nowhere else until now).

>The "proper"™ solution to those design flaws was implemented in Mesos

And yet somehow Mesos failed... sigh.

Re: Nomad vs. Kubernetes

#342

Post by busterarm »

jpgvm wrote: post #324
busterarm wrote: post #246

Earlier quoted context omitted.

As someone with 7-digit spend in GKE/EKS, I will agree with you that it is _anything but simple_. Your developers aren't going to say that it's simple when Google force upgrades their cluster to a version that deprecates APIs in their yamls for a job they worked on 2 years ago and swiftly forgot about. Then when you explain to them that Google insists on putting everyone on a force-upgrade treadmill, you can literall…

So instead of using higher level primitives that are widely used and tested and have many simple high quality integrations (external-dns, cert-manager, et al) you would recommend reinventing all of that on Nomad and then calling that "saving a lot of money and development effort"? Yeah no thanks. At this point k8s has "won" for all intents and purposes. It has gained critical mass, succeeding where other infrastructu…

Thanks, Mr. Well Ackshually,

First of all, those things you're talking about are installables in K8s. They don't come by default. Many people with (managed) kubernetes installations aren't even using them and get by just fine. Having them certainly isn't free and work certainly had to be done to build those for Kubernetes and likely are (or will be) trivial to implement elsewhere. I certainly was able to automate my certificate management infrastructure before I had Kubernetes.

The reality is that there are many companies out there that want like 10% of the features of an orchestrator like Kubernetes and don't need all of those features that make you think that it's a zero-sum game that Kubernetes has won.

The reality is that there are competing offers like Nomad that more-easily accomplish our technology goals and thus more attractive to enterprises with big budgets like mine.

Kuberenetes having the "critical-mass" that you speak of isn't to the exclusion of other competing tools having "critical-mass". Just like Oracle and DB2 aren't the dominant RDBMS of today.

As for calling people indulgent for using beta APIs, let's not forget that Deployment, StatefulSet, DaemonSet, ReplicaSet, NetworkPolicy and PodSecurityPolicy all started as beta APIs and were only removed in 1.16. And I imagine you wanted to use ingresses, but the old ingress.class annotation was deprecated and replaced with IngressClass in 1.18 and that was firm cutover for _everyone_. I can't imagine Kubernetes being very useful to many people without any of these...

Re: Nomad vs. Kubernetes

#343

Post by websap »

shaklee3 wrote: post #309
websap wrote: post #238

Earlier quoted context omitted.

I couldn't disagree with this more. I manage and run Kubernetes clusters as part of my job, and I can tell you that configuring, running and installing these clusters is no small feat. I don't know much about Nomad, but I would highly encourage most users to not think K8S is simple by any standard. Multiple cloud providers now provide ways to run your code directly, or your containers directly. Unless you have a reas…

I also maintained several kubernetes clusters and never found it very difficult. We never had any major outages, and the only downtime was trying new features.

What's the scale of your clusters? If you're running anywhere between 5 - 20 nodes with a 10 pods on each node. I think that's a very small deployment and you'll be able to breeze by with most of the defaults. You still need to configure the following though - logs, certificates, authentication, authorization, os upgrades, kubernetes upgrades, etc.

I'm not sure if all of this is worth it, if you're running a small footprint. You're better of using more managed solutions available today.

Re: Nomad vs. Kubernetes

#344

Post by still_grokking »

ksec wrote: post #341
still_grokking wrote: post #300

Earlier quoted context omitted.

Actually Google knew at the time they "designed" k8s that Borg doesn't scale due to fundamental design flaws in it's basic architecture. Still they reused the exact same architecture for k8s. Who wants to know the details of those scaling issues can google for the Omega paper. The "proper"™ solution to those design flaws was implemented in Mesos (and to my knowledge nowhere else until now).

>The "proper"™ solution to those design flaws was implemented in Mesos And yet somehow Mesos failed... sigh.

Because the market never chooses the most advanced technology.

It chooses the (perceived) cheapest thing with the best marketing. Always.

Re: Nomad vs. Kubernetes

#345

Post by shaklee3 »

websap wrote: post #343
shaklee3 wrote: post #309

Earlier quoted context omitted.

I also maintained several kubernetes clusters and never found it very difficult. We never had any major outages, and the only downtime was trying new features.

What's the scale of your clusters? If you're running anywhere between 5 - 20 nodes with a 10 pods on each node. I think that's a very small deployment and you'll be able to breeze by with most of the defaults. You still need to configure the following though - logs, certificates, authentication, authorization, os upgrades, kubernetes upgrades, etc. I'm not sure if all of this is worth it, if you're running a small fo…

5 clusters of around 100 nodes each on average

Re: Nomad vs. Kubernetes

#346

Post by gilbetron »

ksec wrote: post #340
gilbetron wrote: post #331

Earlier quoted context omitted.

What are you even talking about? Plenty of languages were around then and C dominated all of them because it let people write great software.

I am sure a few old HNer will tell you, C dominated because of UNIX, and the rest is history.

I am old, and that is not the case (although Unix was one of the reasons).

Re: Nomad vs. Kubernetes

#347

Post by AlexCoventry »

rixed wrote: post #316
AlexCoventry wrote: post #305

Earlier quoted context omitted.

That surprises me. Does Google have a more complete secrets-management system for its in-house services?

IIUC, despite K8s having started at Google by Go enthusiasts who had good knowledge of borg, the goal has never been to write a borg clone, even less a replacement for borg. And after so many years of independent development, I see no reason to believe that K8s ressemble borg any more than superficially. This seems to be very much assumed by kubernetes authors. Current borg users please correct me if I'm wrong.

Thanks.

Re: Nomad vs. Kubernetes

#348

Post by ablekh »

jnsaff2 wrote: post #248
q3k wrote: post #169

Earlier quoted context omitted.

There's two things: 1) Text templating YAML is just bad. It's the serialized format of some structured data - instead of templating its text representation (and dealing with quoting, nindent, stringly-typed variables, and general YAML badness), just manipulate the structures directly before serializing them. For example, write some Python code that composes these structures in memory and then just serializes them to…

That is a very helpful summary. I generally favor declarative configuration (like CloudFormation) and I think it's mostly due to my work being infrastructure focused: "I need VPC, LB's, ASG with those immutable Images, a queue, buckets etc". But in my recent work with a customer where the infrastructure is "EKS with datastores, and Gitlab CI" .. most of the complexity is people creating abstractions on top of abstrac…

> Something that would be like CDK for k8s could actually be amazingly useful.

Such thing surely does exist: https://cdk8s.io. And here's the relevant announcement blog post: https://aws.amazon.com/blogs/containers/introducing-cdk-for-....

Re: Nomad vs. Kubernetes

#349

Post by jpgvm »

busterarm wrote: post #342
jpgvm wrote: post #324

Earlier quoted context omitted.

So instead of using higher level primitives that are widely used and tested and have many simple high quality integrations (external-dns, cert-manager, et al) you would recommend reinventing all of that on Nomad and then calling that "saving a lot of money and development effort"? Yeah no thanks. At this point k8s has "won" for all intents and purposes. It has gained critical mass, succeeding where other infrastructu…

Thanks, Mr. Well Ackshually, First of all, those things you're talking about are installables in K8s. They don't come by default. Many people with (managed) kubernetes installations aren't even using them and get by just fine. Having them certainly isn't free and work certainly had to be done to build those for Kubernetes and likely are (or will be) trivial to implement elsewhere. I certainly was able to automate my…

There was no "Well Ackshually" in my reply.

I just pointed out that for everything you would need to build custom on Nomad there exists off-the-shelf components that will plug right into k8s. There is no way this would save money or development time, which was my main contention with your assertions.

You basically mischaracterized everything I said and then missed the entire point.

I don't disagree that there are some cases where Nomad would be superior. Off the top of my head if I wanted to build a modern rendering farm and I knew I wouldn't want to use the cluster for anything else then I would consider it over the HPC toolkits I have previously used for that (RIP Grid Engine) or Mesos which used to be king of that space.

The problem is these are incredibly niche cases and for 99.99% of companies they are better off swimming with the flow.

This is what I mean by "k8s has won". It's not that other things won't continue to exist and be created but most of them won't survive unless they commercialise within a niche or find a sufficiently large user that is willing to do the vast majority of the development (eg. Netflix Titan).

It's not a zero sum game but it's damn close. See here the corpses of Docker Swarm, Convox, Flynn ( :( ), original Deis, arguably ECS (it's a zombie at this point let's be honest).

The tide really turned when Azure and AWS were basically forced into offering managed k8s. In AWS case they also had to made deep modifications to VPC and IAM to properly support it. These investments wouldn't have been made unless forced which lends credence to the weight k8s has in the ecosystem. (Worth mentioning there is no hosted Nomad I'm aware of, their enterprise offering is install + support)

k8s is becoming the POSIX of distributed scheduling. The API you use to run diverse workloads over large numbers of machines that is relatively portable between companies. Right now there is still some vendor nonsense going on but over time it will be smoothed out.

The vast majority of distributed applications are going to be built targeting k8s as their runtime API. We can already see this happening but it will only increase over time.

To summarize I think it's fair to say things like "Nomad can sometimes be good if conditions x/y/z are met" but I think it's very dishonest to consider it as a viable competitor to k8s in the general case for most users because of the reasons outlined above.

Re: Nomad vs. Kubernetes

#350

Post by rstuart4133 »

spookthesunset wrote: post #183
kbenson wrote: post #176

Earlier quoted context omitted.

Probably less calculated and more "that's what's available to offer stably right now that we can feasibly deliver, so that's what we'll do." Distributed RDBMS were not exactly cheap or common in open source a couple decades back. I don't think there was much of a choice to make.

I mean it is a trade off though. You cannot beat the speed of light. The further apart your database servers are, the more lag you get between them. If you want a transactional, consistent datastore you are gonna have to put a lock on something while writes happen. And if you want consistency it means those locks need to be on all systems in the cluster. And the entire cluster needs to hold that lock until the transa…

> If you want a transactional, consistent datastore you are gonna have to put a lock on something while writes happen. And if you want consistency it means those locks need to be on all systems in the cluster.

FWIW, it's not as bad as that sounds. There are traditional locks, and there is optimistic locking. If a there are two conflicting transactions, a traditional lock detects this before it happens (by insisting a lock is obtained before any updates are done) and if there is any chance of conflict the updates are run serially (meaning one is stopped while the other runs).

Optimistic locks let updates run with lock or blocking at all, but then at the end they check if the data they depended on (ie, data that would have been locked by the traditional mechanism) has changed. If it has they throw it all away. (Well, perhaps not quite - they may apply one of the conflicting updates to ensure forward progress is made.) The upside of this is there is if there are no conflicting updates everything runs at full speed - because there is no expensive communication about why has what lock going. The downside is a lot of work may be thrown away by what amounts to speculative execution.

Most monolithic databases use traditional locking. Two CPU's in the same data centre (or more likely on the same board) can rapidly decide who owns what lock, but cycles and I/O on a high end server are precious. Distributed ACID databases like spanner, cockroachdb and yugabytedb favour opportunistic because sending messages half way across the planet to decide who owns what lock before allowing things to proceed takes a lot of time, whereas the CPU cycles and I/O's on the low end replicated hardware are cheap.

While opportunistic locks allow an almost unlimited number of non-conflicting updates to happen concurrently, their clients still have to pay a time penalty. The decision about whether there was a conflicting update still has to be made, and it still requires packets to cross the planet, and while all this happens the client can't be sure if their data has been committed. But unlike the traditional model, they are never blocked by what any other client is doing - providing it doesn't conflict.

Post reply on HN