Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning to operate Kubernetes reliably

stripe.com

Post reply on HN
Print view View original

71–80 of 102 posts

Re: Learning to operate Kubernetes reliably

#71

Post by akvadrako »

alexebird wrote: post #56

I always search for mentions of Hashicorp Nomad in the comments section of front-page Kubernetes articles like this. There are often few or no mentions, so I’d like to add a plug for the Hashistack. For some reason Nomad seems to get noticeably less publicity than some of the other Hashicorp offerings like Consul, Vault, and Terraform. In my opinion Nomad is right up there with them. The documentation is excellent. I…

I never heard of Nomad, but I can't see why I would choose it over the much more popular and standardised k8s.

The biggest benefits seem to be

(1) simplicity, but GCE and minikube are easy enough to learn in a day and

(2) ability to run non-containers, but docker containers are generic - they can run java apps just fine.

Re: Learning to operate Kubernetes reliably

#72

Post by KaiserPro »

ownagefool wrote: post #70
KaiserPro wrote: post #66

Earlier quoted context omitted.

I read the article, I understand completely and I've heard that argument before. Thats why at my company we have three incompatible, half arsed K8s clusters. At the point where you have to fix upstream bugs, its the point where one says: fuckit, its not stable enough, more trouble than its worth. Lets use gaffer tape and move on. As for maintenance, without company buyin for transplanting the _entire_ stack, its ques…

I think you're sort of right but not really. If the benefits of running k8s outweigh the effort of kicking a few patches upstream. Further, if nobody is kicking patches upstream, where exactly are out open source solutions coming from? I would counter argue about the times jenkins has bit me in the ass, but actually, most solutions will when you go deep enough.

Jenkins is an utter utter arse, don't get me wrong. I would gladly pay for circleCI for 90% of usecases. We have > 90 jenkins masters here (don't ask) all in various states of rot. all of them are unceasingly tedious.

However, for getting a script to run on a certain bunch of nodes, at a certain time for given conditions, its pretty simple. (Unless you have a fetish for the myriad of unstable jenkins plugins)

K8s however isn't simple for that usecase. If I had to read the code and then push changes _before_ it worked for my usecase I'd have dropped it like a bag of sick.

However I do take your point that if no one pushes upstream then its not very fun at all.

Re: Learning to operate Kubernetes reliably

#73

Post by tokenizerrr »

alexebird wrote: post #56

I always search for mentions of Hashicorp Nomad in the comments section of front-page Kubernetes articles like this. There are often few or no mentions, so I’d like to add a plug for the Hashistack. For some reason Nomad seems to get noticeably less publicity than some of the other Hashicorp offerings like Consul, Vault, and Terraform. In my opinion Nomad is right up there with them. The documentation is excellent. I…

I'm not sure how Consul is doing nowadays, but when I used it about two years ago I've had nothing but issues.

Re: Learning to operate Kubernetes reliably

#74

Post by zie »

akvadrako wrote: post #71
alexebird wrote: post #56

I always search for mentions of Hashicorp Nomad in the comments section of front-page Kubernetes articles like this. There are often few or no mentions, so I’d like to add a plug for the Hashistack. For some reason Nomad seems to get noticeably less publicity than some of the other Hashicorp offerings like Consul, Vault, and Terraform. In my opinion Nomad is right up there with them. The documentation is excellent. I…

I never heard of Nomad, but I can't see why I would choose it over the much more popular and standardised k8s. The biggest benefits seem to be (1) simplicity, but GCE and minikube are easy enough to learn in a day and (2) ability to run non-containers, but docker containers are generic - they can run java apps just fine.

I would argue the biggest strength is maintainability. Managing and keeping up a distributed cluster with k8s is WORK. If you are not at the scale where you can dedicate full-time staff to managing only k8s, you shouldn't even be touching k8s. You need full-time staff to keep it alive.

Nomad is operationally simple, you can run it out of your normal devops roles, you don't need dedicated staff. Mostly because you can pretty easily wrap your head around what it does and how it works.

This saves you bundles of cash and time.

Re: Learning to operate Kubernetes reliably

#75

Post by toong »

tokenizerrr wrote: post #73
alexebird wrote: post #56

I always search for mentions of Hashicorp Nomad in the comments section of front-page Kubernetes articles like this. There are often few or no mentions, so I’d like to add a plug for the Hashistack. For some reason Nomad seems to get noticeably less publicity than some of the other Hashicorp offerings like Consul, Vault, and Terraform. In my opinion Nomad is right up there with them. The documentation is excellent. I…

I'm not sure how Consul is doing nowadays, but when I used it about two years ago I've had nothing but issues.

Having a correct mental modal of the Consul architecture and realizing that the raft cluster (consistency) and the consul cluster (gossip) are two separate layers, does wonders.

Additionally, in the early days there were some tools missing (like online modifying the raft peer members) that are all there now.

Running in production and very happy with it!

Re: Learning to operate Kubernetes reliably

#76

Post by kvz »

alexebird wrote: post #56

I always search for mentions of Hashicorp Nomad in the comments section of front-page Kubernetes articles like this. There are often few or no mentions, so I’d like to add a plug for the Hashistack. For some reason Nomad seems to get noticeably less publicity than some of the other Hashicorp offerings like Consul, Vault, and Terraform. In my opinion Nomad is right up there with them. The documentation is excellent. I…

Being a bit of a HashiCorp fan I tried Nomad for Transloadit but at the time it did not support persistent volumes. K8s had that already. The more I started looking into k8s as an alternative, the more compelling features I discovered that Nomad did not have yet.

With the velocity of k8s it's hard to imagine how Nomad could catch/keep up. K8s has operators, Helm, etc. That just means you can add battle-tested components off the shelve with a single command. So, less wheel-inventing and boilerplate writing to do for us.

With the backing of so much larger community/entities it also feels like I’m less likely to be the first one to discover a new bug. RedHat or Google or one of their customers will have hit and fixed it already, and my production platform keeps humming along nicely. K8s has just had more flytime and exposure to crazy environments and workloads, so more kinks are going to be ironed out.

I always did like the “do one thing right” unixy approach of Hashicorp’s toolset, and that you can pick the pieces you like. But (sadly for them) that means I can now pick Vault or Consul and run it on top of Kubernetes (re-using k8s' internal etcd is not recommended) if I wanted. I'm actually not overly sorry for them, seeing as how they're locking up more & more features behind enterprise products. I haven't checked in a while but wouldn't be surprised if they also had a Nomad Enterprise already. Nothing wrong with HashiCorp wanting to make money, but if there also is k8s without those restrictions..

Re: Learning to operate Kubernetes reliably

#77

Post by akvadrako »

zie wrote: post #74
akvadrako wrote: post #71

Earlier quoted context omitted.

I never heard of Nomad, but I can't see why I would choose it over the much more popular and standardised k8s. The biggest benefits seem to be (1) simplicity, but GCE and minikube are easy enough to learn in a day and (2) ability to run non-containers, but docker containers are generic - they can run java apps just fine.

I would argue the biggest strength is maintainability. Managing and keeping up a distributed cluster with k8s is WORK. If you are not at the scale where you can dedicate full-time staff to managing only k8s, you shouldn't even be touching k8s. You need full-time staff to keep it alive. Nomad is operationally simple, you can run it out of your normal devops roles, you don't need dedicated staff. Mostly because you can…

I don't see why - I have my GCE cluster running fine with zero maintenance work.

Re: Learning to operate Kubernetes reliably

#78

Post by jnsaff2 »

alexebird wrote: post #56

I always search for mentions of Hashicorp Nomad in the comments section of front-page Kubernetes articles like this. There are often few or no mentions, so I’d like to add a plug for the Hashistack. For some reason Nomad seems to get noticeably less publicity than some of the other Hashicorp offerings like Consul, Vault, and Terraform. In my opinion Nomad is right up there with them. The documentation is excellent. I…

I have a few production Mesos clusters under my belt and one production Nomad and I really like Nomad and Mesos is not bad.

Kubernetes seems to be a lot of magic and NIH and tries to do everything itself, whereas Mesos and Nomad are nicely composable and easy to reason about.

Nomad's biggest benefit for me is a very nice integration with Vault (and Consul), I can have Nomad ask for a container instance specific secret which Vault then goes and generates and later immediately revokes once that container dies. Maybe this is possible with Kubernetes but I have not seen anything that tight yet.

IAM instance profiles are nice but they are instance wide, but having each container a unique, short lived and properly scoped set of secrets injected at the last possible time and immediately revoked afterwards makes me feel all warm and fuzzy inside.

Re: Learning to operate Kubernetes reliably

#79

Post by karussell »

thesandlord wrote: post #17
013a wrote: post #14

Earlier quoted context omitted.

I would argue that, while Stripe is going with a scratch build, this could be motived by AWS's lack of a good managed Kube offering, which is changing in the next few months. With a managed Kube offering, setting up Kube is much much easier than this jenkins setup you are suggesting. And, there's no overhead charge. Why would anyone go through the hassle of manually provisioning machines like you suggest when AWS/GCP…

(I work for GCP) For 95% of people, I'd say going with the managed version is the right choice. However there are some reasons why you wouldn't use a managed service. If you need a custom build, custom drivers, etc.

Do you think GKE will support multi clouds setups or hybrid scenarios at some point? For cost reasons we have to put some big servers off the cloud ...

Re: Learning to operate Kubernetes reliably

#80

Post by erkkie »

akvadrako wrote: post #71
alexebird wrote: post #56

I always search for mentions of Hashicorp Nomad in the comments section of front-page Kubernetes articles like this. There are often few or no mentions, so I’d like to add a plug for the Hashistack. For some reason Nomad seems to get noticeably less publicity than some of the other Hashicorp offerings like Consul, Vault, and Terraform. In my opinion Nomad is right up there with them. The documentation is excellent. I…

I never heard of Nomad, but I can't see why I would choose it over the much more popular and standardised k8s. The biggest benefits seem to be (1) simplicity, but GCE and minikube are easy enough to learn in a day and (2) ability to run non-containers, but docker containers are generic - they can run java apps just fine.

One huge benefit of nomad is that it can schedule non containers too, enabling fixing up legacy systems incrementally.
Post reply on HN