Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning to operate Kubernetes reliably

stripe.com

Post reply on HN
Print view View original

31–40 of 102 posts

Re: Learning to operate Kubernetes reliably

#31

Post by dominotw »

antoncohen wrote: post #21
KaiserPro wrote: post #4

Much as it burns me to admit this, for this usecase, jenkins is king. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user…

I disagree that Jenkins is king for this. Jenkins is a single point of failure, is isn't a highly available distributed scheduler. It is a single master with slaves. While it is easy to configure Jenkins jobs with code (Job Builder, Job DSL, Jenkinsfiles), it is a pain to manage Jenkins itself with code. Plugins, authentication, all the non-job configuration, that is usually done via the GUI. Saying Jenkins can be co…

> it is a pain to manage Jenkins itself with code. Plugins, authentication, all the non-job configuration, that is usually done via the GUI.

This is not true, all the configuration is scriptable via groovy scripts. We run bunch of groovy startup scripts that configure everything post launch. There is an effort to support this better[1] by jenkins team.

> How to you isolate job executions in Jenkins? In Kubernetes each job inherently isolated in containers.

We run one docker container/build on docker swarm. Each build gets its own isolated/clean environment. There is no EC2 provisioning ect. We already own and maintain docker swarm setup we just run jenkins/jenkins agents on it. I assume if you are using kubernetes it would be similar setup.

> Jenkins is a single point of failure, is isn't a highly available distributed scheduler.

I agree with this to an extent. If you are running jenkins on scheduler it can be rescheduled but you inflight jobs are dead.

1. https://github.com/jenkinsci/configuration-as-code-plugin

Re: Learning to operate Kubernetes reliably

#32

Post by pishpash »

robszumski wrote: post #20

> “Sometimes when we do an etcd failover, the API server starts timing out requests until we restart it.” This is likely related a set of Kubernetes bugs [1][2] (and grpc[3]) that CoreOS is working diligently to get fixed. The first set of these, the endpoint reconciler[4], has landed in 1.9. More work is pending on the etcd client in Kubernetes. The good news is that the client is used everywhere, so one fix and all…

I don't get this. Didn't Kubernetes come out of Google Borg that had been in use forever? The second write should be more elegant and impressive -- why so many basic bugs?

Re: Learning to operate Kubernetes reliably

#33

Post by ecnahc515 »

user5994461 wrote: post #18
numbsafari wrote: post #16

Earlier quoted context omitted.

I've also previously used jenkins for cron to pretty good effect (I like to call it "jcron"). The ability to define jobs in yaml and have it be driven from your scm is really awesome. However, k8s does more than just scheduling where pods run. It also ensures that they run with the correct security and availability constraints. When you add in things like affinity (don't run this job on the same machine as that job,…

Jenkins also has a notion of hosts and tags to decide where jobs are assigned.

Though it's resource awareness is lacking, which is where k8s shines. Honestly I find combining Jenkins and K8s a relatively pleasant experience. The jenkins kubernetes-plugin has gaps and issues, but with time it will mature. There's no reason you can't combine them to get the best of both worlds.

Re: Learning to operate Kubernetes reliably

#34

Post by ecnahc515 »

pishpash wrote: post #32
robszumski wrote: post #20

> “Sometimes when we do an etcd failover, the API server starts timing out requests until we restart it.” This is likely related a set of Kubernetes bugs [1][2] (and grpc[3]) that CoreOS is working diligently to get fixed. The first set of these, the endpoint reconciler[4], has landed in 1.9. More work is pending on the etcd client in Kubernetes. The good news is that the client is used everywhere, so one fix and all…

I don't get this. Didn't Kubernetes come out of Google Borg that had been in use forever? The second write should be more elegant and impressive -- why so many basic bugs?

The concepts are based on similar ideas in Borg, but the things it shares with Borg ends there.

The implementation is effectively entirely from scratch, so bugs will exist.

Re: Learning to operate Kubernetes reliably

#35

Post by jvns »

ad_hominem wrote: post #27

How do you deal with sidecar containers in CronJobs (and regular batch Jobs) not terminating correctly? https://github.com/kubernetes/kubernetes/issues/25908

We don't run sidecar containers in cron jobs yet. That said, here's a workaround (from that issue): https://github.com/kubernetes/kubernetes/issues/25908#issuec...

Re: Learning to operate Kubernetes reliably

#36

Post by ianstormtaylor »

KaiserPro wrote: post #4

Much as it burns me to admit this, for this usecase, jenkins is king. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user…

It feels like you didn’t read the article.

The author made clear multiple times that they were using cron jobs as a test bed for Kubernetes, and they chose to “overengineer” because they’re looking to use Kubernetes for more and more of their needs over time. You’re kind of arguing against a straw man.

I think it’s actually a great example of how Stripe thinks about technology choices.

They’re interested in choosing fewer tools that are better built and can grow to solve more needs. And they’re evaluating tools not just by “time to complete X random project”, but by other longer-term heuristics like maintenance levels. And the best way to do that is to start using the tool for a single need, investing more time in learning/research than is required for the need itself—ensuring that it really is a solid, foundational solution—with the understanding that you’re choosing technology for the long run. Then continue to expand your use of the tool over time, reaping benefits on your initial time investment.

Re: Learning to operate Kubernetes reliably

#37

Post by ad_hominem »

jvns wrote: post #35
ad_hominem wrote: post #27

How do you deal with sidecar containers in CronJobs (and regular batch Jobs) not terminating correctly? https://github.com/kubernetes/kubernetes/issues/25908

We don't run sidecar containers in cron jobs yet. That said, here's a workaround (from that issue): https://github.com/kubernetes/kubernetes/issues/25908#issuec...

I'm aware of the workarounds in that thread. Just wondering if Stripe had a different workaround but I guess not.

Re: Learning to operate Kubernetes reliably

#38

Post by rsanders »

pishpash wrote: post #32
robszumski wrote: post #20

> “Sometimes when we do an etcd failover, the API server starts timing out requests until we restart it.” This is likely related a set of Kubernetes bugs [1][2] (and grpc[3]) that CoreOS is working diligently to get fixed. The first set of these, the endpoint reconciler[4], has landed in 1.9. More work is pending on the etcd client in Kubernetes. The good news is that the client is used everywhere, so one fix and all…

I don't get this. Didn't Kubernetes come out of Google Borg that had been in use forever? The second write should be more elegant and impressive -- why so many basic bugs?

Kubernetes didn't come with all of the other Google infrastructure.

Re: Learning to operate Kubernetes reliably

#39

Post by scrollaway »

KaiserPro wrote: post #4

Much as it burns me to admit this, for this usecase, jenkins is king. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user…

Been using Jenkins a bunch here and cronjobs are the only thing it does really nicely. We're thinking of switching to CircleCI for builds though (which has been a pain because no self-hosting), and I'm not sure Jenkins makes sense to keep as only a cronjobber.

Has anyone used Airflow for cronjobs? is it a good idea or a terrible one?

Re: Learning to operate Kubernetes reliably

#40

Post by eric_b »

jvns wrote: post #13
KaiserPro wrote: post #4

Much as it burns me to admit this, for this usecase, jenkins is king. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user…

Hi! Post author here! I agree that it's really important to be careful of "shiny new tool" syndrome -- one of my primary goals in writing this post was to show that operating Kubernetes in production is complicated and to encourage people to think carefully before introducing a Kubernetes cluster into their infrastructure. As you say -- I think by itself "we want to run some cron jobs" isn't a good enough reason by i…

> A goal for this project was to prove to ourselves that we actually could run production code in Kubernetes, to learn about how much work operating Kubernetes actually is, and to lay the groundwork for moving more things to Kubernetes in the future.

Why wasn't the final sentence "and to re-evaluate if moving forward was even a good idea?"

Because I get nervous every time someone is relying on their patches to be included upstream. Or they need to dive in to the internals of something repeatedly. That screams "not production ready" to me.

After reading the post, Kubernetes did not sound at all like a slam dunk in terms of a solution, let alone a foundation for more mission critical infrastructure. The Jenkins solution offered by the parent sounds more reasonable, even with the objections you list.

Edit: Take my comments with a grain of salt, but from internet armchair vantage point it does sound like Kubernetes was chosen first, and rationalized second. (Though I very much appreciated the thoroughness with which you went about learning the technology)

Post reply on HN