Setting aside the k8s content itself, I love the way this article is written. It's not a typical tutorial or tips/tricks but takes you time-traveling through the experience of a big company adopting nascent tech. Lot of great things to take away even outside of the kubernetes tips.
Learning to operate Kubernetes reliably
11–20 of 102 posts
Re: Learning to operate Kubernetes reliably
#12Kubernetes very recently added native Cronjob support: https://kubernetes.io/docs/concepts/workloads/controllers/cr... How does Stripe's approach differ?
No difference — we are using Kubernetes's native cronjob support. This post is about how we migrated to that system.
Re: Learning to operate Kubernetes reliably
#13Much as it burns me to admit this, for this usecase, jenkins is king. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user…
Hi! Post author here! I agree that it's really important to be careful of "shiny new tool" syndrome -- one of my primary goals in writing this post was to show that operating Kubernetes in production is complicated and to encourage people to think carefully before introducing a Kubernetes cluster into their infrastructure.
As you say -- I think by itself "we want to run some cron jobs" isn't a good enough reason by itself to use Kubernetes (though it might be a good enough reason if you’re using a managed Kubernetes cluster where someone else handles the cluster operations). A goal for this project was to prove to ourselves that we actually could run production code in Kubernetes, to learn about how much work operating Kubernetes actually is, and to lay the groundwork for moving more things to Kubernetes in the future.
In my mind, a huge advantage of Kubernetes is that Kubernetes' code is very readable and they're great at accepting contributions. In the past when we've run into performance problems with Jenkins (we also use jenkins-job-builder to manage our 1k node Jenkins cluster), they've been extremely difficult to debug and it's hard to get visibility into what's going on inside Jenkins. I find Kubernetes’ code a lot easier to read, it's fairly easy to monitor the internals, and the core components have pprof included by default if you want to get profiling information out. Being able to easily fix bugs in Kubernetes and get the patches merged upstream has been a big deal for us.
Re: Learning to operate Kubernetes reliably
#14Much as it burns me to admit this, for this usecase, jenkins is king. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user…
I would argue that, while Stripe is going with a scratch build, this could be motived by AWS's lack of a good managed Kube offering, which is changing in the next few months.
With a managed Kube offering, setting up Kube is much much easier than this jenkins setup you are suggesting. And, there's no overhead charge. Why would anyone go through the hassle of manually provisioning machines like you suggest when AWS/GCP will do it for you?
Its overkill in the same way using DynamoDB for something that only experiences a handful of writes every day is overkill; who cares? The scale is there if you need it, but it doesn't cost anything to not use it.
Re: Learning to operate Kubernetes reliably
#15Much as it burns me to admit this, for this usecase, jenkins is king. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user…
Why not both? - https://github.com/jenkinsci/kubernetes-plugin
You provide a scalable infrastructure underneath your jenkins install while not dealing with the issue of node/agent allocation. Plus, you get kubernetes if for your not-so-simple crons.
Re: Learning to operate Kubernetes reliably
#16Much as it burns me to admit this, for this usecase, jenkins is king. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user…
I've also previously used jenkins for cron to pretty good effect (I like to call it "jcron"). The ability to define jobs in yaml and have it be driven from your scm is really awesome.
However, k8s does more than just scheduling where pods run. It also ensures that they run with the correct security and availability constraints. When you add in things like affinity (don't run this job on the same machine as that job, or, only run jobs for this tenant on nodes assigned to that tenant), storage management (connect this job to this volume), networking (only let this pod talk to this service and the monitoring layer, don't let anyone connect to the pods running the job), and much, much more.
Yeah, you can do that with jenkins, or like, just cron. I know, because I did it for 18 years before I had ever heard of Kubernetes.
But, just like I can reach for Django or Rails or whatever it is that Java programmers use these days to build my web application, I can lean on Kubernetes to build my infrastructure.
I estimate that leveraging GKE has saved me in the range of $400k in direct employee costs, not to mention time-to-market advantages. As we grow, I expect that number to go higher.
Re: Learning to operate Kubernetes reliably
#17Much as it burns me to admit this, for this usecase, jenkins is king. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user…
I would argue that, while Stripe is going with a scratch build, this could be motived by AWS's lack of a good managed Kube offering, which is changing in the next few months. With a managed Kube offering, setting up Kube is much much easier than this jenkins setup you are suggesting. And, there's no overhead charge. Why would anyone go through the hassle of manually provisioning machines like you suggest when AWS/GCP…
(I work for GCP)
For 95% of people, I'd say going with the managed version is the right choice.
However there are some reasons why you wouldn't use a managed service. If you need a custom build, custom drivers, etc.
Re: Learning to operate Kubernetes reliably
#18Much as it burns me to admit this, for this usecase, jenkins is king. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user…
I've also previously used jenkins for cron to pretty good effect (I like to call it "jcron"). The ability to define jobs in yaml and have it be driven from your scm is really awesome. However, k8s does more than just scheduling where pods run. It also ensures that they run with the correct security and availability constraints. When you add in things like affinity (don't run this job on the same machine as that job,…
Jenkins also has a notion of hosts and tags to decide where jobs are assigned.
Re: Learning to operate Kubernetes reliably
#19Earlier quoted context omitted.
I would argue that, while Stripe is going with a scratch build, this could be motived by AWS's lack of a good managed Kube offering, which is changing in the next few months. With a managed Kube offering, setting up Kube is much much easier than this jenkins setup you are suggesting. And, there's no overhead charge. Why would anyone go through the hassle of manually provisioning machines like you suggest when AWS/GCP…
(I work for GCP) For 95% of people, I'd say going with the managed version is the right choice. However there are some reasons why you wouldn't use a managed service. If you need a custom build, custom drivers, etc.
And most importantly, to use across multiple clouds!
Re: Learning to operate Kubernetes reliably
#20> “Sometimes when we do an etcd failover, the API server starts timing out requests until we restart it.”
This is likely related a set of Kubernetes bugs [1][2] (and grpc[3]) that CoreOS is working diligently to get fixed. The first set of these, the endpoint reconciler[4], has landed in 1.9.
More work is pending on the etcd client in Kubernetes. The good news is that the client is used everywhere, so one fix and all components will benefit.
[1]: https://github.com/kubernetes/community/pull/939 [2]: https://github.com/kubernetes/kubernetes/issues/22609 [3]: https://github.com/kubernetes/kubernetes/issues/47131 [4]: https://github.com/kubernetes/kubernetes/pull/51698