Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning to operate Kubernetes reliably

stripe.com

Post reply on HN
Print view View original

51–60 of 102 posts

Re: Learning to operate Kubernetes reliably

#51

Post by vicaya »

asimpletune wrote: post #49
vicaya wrote: post #46

Earlier quoted context omitted.

FTFA: "We’d previously been using Chronos (with Mesos) as a cron job scheduling system, but it was no longer meeting our reliability requirements and it’s mostly unmaintained (1 commit in the last 9 months, and the last time a pull request was merged was March 2016) Because Chronos is unmaintained, we decided it wasn’t worth continuing to invest in improving our existing cluster." Though Chronos has a release recentl…

> Mesos is inevitably fading as a legacy platform. Because of Chronos? This is a bizarre thing to say. Mesos actually works extremely well. Whenever I ask the why kube over Mesos question, I never get a good answer. I think because people just don’t know Mesos. Also it wasn’t made by google.

Chronos is just an example. There're many bugs in Mesos that don't get fixed for months/years. Mesos core is legacy (pre 11) C++ code nobody wants to maintain.

Re: Learning to operate Kubernetes reliably

#52

Post by y4mi »

dominotw wrote: post #31
antoncohen wrote: post #21

Earlier quoted context omitted.

I disagree that Jenkins is king for this. Jenkins is a single point of failure, is isn't a highly available distributed scheduler. It is a single master with slaves. While it is easy to configure Jenkins jobs with code (Job Builder, Job DSL, Jenkinsfiles), it is a pain to manage Jenkins itself with code. Plugins, authentication, all the non-job configuration, that is usually done via the GUI. Saying Jenkins can be co…

> it is a pain to manage Jenkins itself with code. Plugins, authentication, all the non-job configuration, that is usually done via the GUI. This is not true, all the configuration is scriptable via groovy scripts. We run bunch of groovy startup scripts that configure everything post launch. There is an effort to support this better[1] by jenkins team. > How to you isolate job executions in Jenkins? In Kubernetes eac…

Since when does docker swarm support build steps pre launch?

Are you sure you're not just using plain docker on docker swarm nodes?

Re: Learning to operate Kubernetes reliably

#53

Post by RA_Fisher »

eric_b wrote: post #40
jvns wrote: post #13

Earlier quoted context omitted.

Hi! Post author here! I agree that it's really important to be careful of "shiny new tool" syndrome -- one of my primary goals in writing this post was to show that operating Kubernetes in production is complicated and to encourage people to think carefully before introducing a Kubernetes cluster into their infrastructure. As you say -- I think by itself "we want to run some cron jobs" isn't a good enough reason by i…

> A goal for this project was to prove to ourselves that we actually could run production code in Kubernetes, to learn about how much work operating Kubernetes actually is, and to lay the groundwork for moving more things to Kubernetes in the future. Why wasn't the final sentence "and to re-evaluate if moving forward was even a good idea?" Because I get nervous every time someone is relying on their patches to be inc…

Julia is a visibility pro. When things scale, you need to be able to look inside the thing. If that's tough :grimmacing: for probably hundreds of developers. What a waste! /irony

Re: Learning to operate Kubernetes reliably

#54

Post by pm90 »

wmf wrote: post #42
scarface74 wrote: post #30

I'm curious about what people think about HashiCorp's Nad bs Kubernetes. I chose Nomad because I'm already using Consul and I wanted to run raw .Net executables. Would it have been worth it to use Docker with .Net Core? Not trying to change my infrastructure now, but just curious about whether it is worth the time to play with it on the side.

Nomad appears to be better designed, more scalable, and easier to operate than k8s, but it will fall behind pretty rapidly since k8s has 100x more developers.

That isn't necessarily true (playing devils advocate): OpenStack had gajillion developers and still failed (mostly).

Although k8s does seem to be designed much better. I use it personally too and hope for its success.

Re: Learning to operate Kubernetes reliably

#55

Post by pm90 »

vicaya wrote: post #51
asimpletune wrote: post #49

Earlier quoted context omitted.

> Mesos is inevitably fading as a legacy platform. Because of Chronos? This is a bizarre thing to say. Mesos actually works extremely well. Whenever I ask the why kube over Mesos question, I never get a good answer. I think because people just don’t know Mesos. Also it wasn’t made by google.

Chronos is just an example. There're many bugs in Mesos that don't get fixed for months/years. Mesos core is legacy (pre 11) C++ code nobody wants to maintain.

> Mesos core is legacy (pre 11) C++ code nobody wants to maintain.

This is actually very very VERY important. Go is a lot more concise (IMO) that C++, generally when I'm curious about how something works in a project written in Go, its much easier to follow the logic.

Re: Learning to operate Kubernetes reliably

#56

Post by alexebird »

I always search for mentions of Hashicorp Nomad in the comments section of front-page Kubernetes articles like this. There are often few or no mentions, so I’d like to add a plug for the Hashistack.

For some reason Nomad seems to get noticeably less publicity than some of the other Hashicorp offerings like Consul, Vault, and Terraform. In my opinion Nomad is right up there with them. The documentation is excellent. I haven’t had to fix any upstream issues in about a year of development on two separate Nomad clusters. Upgrading versions live is straightforward, and I rarely find myself in a situation where I can’t accomplish something I envisioned because Nomad is missing a feature. It schedules batch jobs, cron jobs, long running services, and system services that run on every node. It has a variety of job drivers outside of Docker.

Nomad, Consul, Vault, and the Consul-aware Fabio load balancer run together to form most of what one might need for a cluster scheduler based deployment, somewhat reminiscent of the “do one thing well” Unix philosophy of composability.

Certainly it isn’t perfect, but I’d recommend it to anyone who is considering using a cluster scheduler but is apprehensive about the operational complexity of the more widely discussed options such as Kubernetes.

Re: Learning to operate Kubernetes reliably

#57

Post by vesak »

asimpletune wrote: post #49
vicaya wrote: post #46

Earlier quoted context omitted.

FTFA: "We’d previously been using Chronos (with Mesos) as a cron job scheduling system, but it was no longer meeting our reliability requirements and it’s mostly unmaintained (1 commit in the last 9 months, and the last time a pull request was merged was March 2016) Because Chronos is unmaintained, we decided it wasn’t worth continuing to invest in improving our existing cluster." Though Chronos has a release recentl…

> Mesos is inevitably fading as a legacy platform. Because of Chronos? This is a bizarre thing to say. Mesos actually works extremely well. Whenever I ask the why kube over Mesos question, I never get a good answer. I think because people just don’t know Mesos. Also it wasn’t made by google.

Or perhaps you do get good answers but you choose to forget them, because they don't quite suit your situation.

Re: Learning to operate Kubernetes reliably

#58

Post by jmillikin »

ad_hominem wrote: post #37
jvns wrote: post #35

Earlier quoted context omitted.

We don't run sidecar containers in cron jobs yet. That said, here's a workaround (from that issue): https://github.com/kubernetes/kubernetes/issues/25908#issuec...

I'm aware of the workarounds in that thread. Just wondering if Stripe had a different workaround but I guess not.

That GitHub comment is Stripe's workaround! I copied it nearly as-is from our internal job setup boilerplate.

Re: Learning to operate Kubernetes reliably

#59

Post by asimpletune »

vesak wrote: post #57
asimpletune wrote: post #49

Earlier quoted context omitted.

> Mesos is inevitably fading as a legacy platform. Because of Chronos? This is a bizarre thing to say. Mesos actually works extremely well. Whenever I ask the why kube over Mesos question, I never get a good answer. I think because people just don’t know Mesos. Also it wasn’t made by google.

Or perhaps you do get good answers but you choose to forget them, because they don't quite suit your situation.

I'd be interested to hear if you have one. I'm serious, I really just want to understand better, not trying to be controversial. Also, "it's not written in go" doesn't count, although I do like go a lot.

Re: Learning to operate Kubernetes reliably

#60

Post by takeda »

ianstormtaylor wrote: post #36
KaiserPro wrote: post #4

Much as it burns me to admit this, for this usecase, jenkins is king. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user…

It feels like you didn’t read the article. The author made clear multiple times that they were using cron jobs as a test bed for Kubernetes, and they chose to “overengineer” because they’re looking to use Kubernetes for more and more of their needs over time. You’re kind of arguing against a straw man. I think it’s actually a great example of how Stripe thinks about technology choices. They’re interested in choosing…

If you have a hammer ...
Post reply on HN