Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning to operate Kubernetes reliably

stripe.com

Post reply on HN
Print view View original

61–70 of 102 posts

Re: Learning to operate Kubernetes reliably

#61

Post by asimpletune »

vicaya wrote: post #51
asimpletune wrote: post #49

Earlier quoted context omitted.

> Mesos is inevitably fading as a legacy platform. Because of Chronos? This is a bizarre thing to say. Mesos actually works extremely well. Whenever I ask the why kube over Mesos question, I never get a good answer. I think because people just don’t know Mesos. Also it wasn’t made by google.

Chronos is just an example. There're many bugs in Mesos that don't get fixed for months/years. Mesos core is legacy (pre 11) C++ code nobody wants to maintain.

This whole thread reads like it's from n-gate.

Re: Learning to operate Kubernetes reliably

#62

Post by asimpletune »

pm90 wrote: post #55
vicaya wrote: post #51

Earlier quoted context omitted.

Chronos is just an example. There're many bugs in Mesos that don't get fixed for months/years. Mesos core is legacy (pre 11) C++ code nobody wants to maintain.

> Mesos core is legacy (pre 11) C++ code nobody wants to maintain. This is actually very very VERY important. Go is a lot more concise (IMO) that C++, generally when I'm curious about how something works in a project written in Go, its much easier to follow the logic.

Go is nice. I like it a lot. It's very readable, it reduces the number of good ways you can do something to usually just one, it's fast, fat binaries are awesome, great concurrency primitives, etc...

Re: Learning to operate Kubernetes reliably

#63

Post by jmillikin »

eric_b wrote: post #40
jvns wrote: post #13

Earlier quoted context omitted.

Hi! Post author here! I agree that it's really important to be careful of "shiny new tool" syndrome -- one of my primary goals in writing this post was to show that operating Kubernetes in production is complicated and to encourage people to think carefully before introducing a Kubernetes cluster into their infrastructure. As you say -- I think by itself "we want to run some cron jobs" isn't a good enough reason by i…

> A goal for this project was to prove to ourselves that we actually could run production code in Kubernetes, to learn about how much work operating Kubernetes actually is, and to lay the groundwork for moving more things to Kubernetes in the future. Why wasn't the final sentence "and to re-evaluate if moving forward was even a good idea?" Because I get nervous every time someone is relying on their patches to be inc…

Hello! I work at Stripe and helped with some aspects of the Kubernetes cron stuff -- maybe these answers can be helpful.

  > Why wasn't the final sentence "and to re-evaluate if
  > moving forward was even a good idea?"
I think that's sort of implied -- complex technical projects have a risk of unexpected roadblocks, and it's important that "stop and roll back" always be on the list of options. Never burn your ships.

We invested a (proportionally) large amount of engineering effort to ensure we had the ability to move the whole shebang back to Chronos ~immediately. As noted in the article, we exercised this rollback feature several times when particular cronjobs deviated from expected behavior when run in Kubernetes.

  > Because I get nervous every time someone is relying on
  > their patches to be included upstream. Or they need to
  > dive in to the internals of something repeatedly. That
  > screams "not production ready" to me.
This is the same basic model as disto-specific patches to the Linux kernel.

Every engineering organization reaches the point where they want more features than are available in an existing platform. The most practical solutions for this are to launch a new platform ("Not Invented Here"), or contribute code upstream. The first option can provide better short-term outcomes, but is usually inferior on multi-year timescales.

Consider that with a mature build infrastructure, internal builds are actually the latest stable release plus cherry-picked patches. This provides the best of all worlds -- an upstream foundation, with bug fixes on our schedule, and an eventually-consistent contribution to the community.

Re: Learning to operate Kubernetes reliably

#64

Post by bonesss »

numbsafari wrote: post #16
KaiserPro wrote: post #4

Much as it burns me to admit this, for this usecase, jenkins is king. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user…

I've also previously used jenkins for cron to pretty good effect (I like to call it "jcron"). The ability to define jobs in yaml and have it be driven from your scm is really awesome. However, k8s does more than just scheduling where pods run. It also ensures that they run with the correct security and availability constraints. When you add in things like affinity (don't run this job on the same machine as that job,…

> I can lean on Kubernetes to build my infrastructure. ... I estimate that leveraging GKE has saved me [$BigMoney]

I'm very sympathetic to the view that jenkins, or something comparable, is viable and cost effective for a lot of shops if you're looking exclusively at direct project costs.

As you've pointed out, though, as a building block of Enterprise software the ability to scale out in, and across, multiple clouds consistently is an economic and development boon so powerful I don't think one should really be looking at k8s as just a microservice/deployment platform: it's a common environment-ignorant application standard. Picking and choosing per service whether you should be hosting in GKE, AWS, or on-premise, applying federated clusters, recreating whole production environments for dev... It's a gamechanger.

It's totally possible to fire up a new Jenkins solution in EC2, but as of a few weeks ago Kubernetes is click-and-go in all three major cloud providers. It totally reshapes how we're looking at development projects with suppliers, testing, etc, as we can create fictionalized shared versions of our production environment for development, integration, and testing. As an emerging industry wide standard we can demand and expect Kubernetes knowledge from third parties in a way a home-brewed Jenkins setup could never match.

Re: Learning to operate Kubernetes reliably

#65

Post by KaiserPro »

antoncohen wrote: post #21
KaiserPro wrote: post #4

Much as it burns me to admit this, for this usecase, jenkins is king. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user…

I disagree that Jenkins is king for this. Jenkins is a single point of failure, is isn't a highly available distributed scheduler. It is a single master with slaves. While it is easy to configure Jenkins jobs with code (Job Builder, Job DSL, Jenkinsfiles), it is a pain to manage Jenkins itself with code. Plugins, authentication, all the non-job configuration, that is usually done via the GUI. Saying Jenkins can be co…

> isn't a highly available distributed scheduler.

Bingo! thats the point, its a cron replacement.

But to tackle your first point, K8s might be distributed, its not inherently reliable. Yeah sure people run it in production, but there are a myriad of bugs that you bump into. I've lost clusters due to tiny issues that ran rampant. Something that I've not had in other cluster or grid engine systems.

if we are talking AWS, then having the jenkins master in an auto scaling group with decent monitoring sorts out most of your uptime issues,

The reason I say it'd take a day to configure jenkins is because the jobs have already been setup in cronos. It should literally be a copy-pasta job. All the hard work of figuring out which jobs are box killers, which can share, which are a bit sticky has been done already, all thats changing is the execution system.

What level of isolation are you after, and for what purpose? if jobs can't live on the same box, then thats almost certainly bad job design. (yes there are exceptions, but unbounded memory or CPU usage is just nasty.) There maybe need for regulatory isolation, but containers are not currently recognised as isolated for that purpose.

Re: Learning to operate Kubernetes reliably

#66

Post by KaiserPro »

ianstormtaylor wrote: post #36
KaiserPro wrote: post #4

Much as it burns me to admit this, for this usecase, jenkins is king. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user…

It feels like you didn’t read the article. The author made clear multiple times that they were using cron jobs as a test bed for Kubernetes, and they chose to “overengineer” because they’re looking to use Kubernetes for more and more of their needs over time. You’re kind of arguing against a straw man. I think it’s actually a great example of how Stripe thinks about technology choices. They’re interested in choosing…

I read the article, I understand completely and I've heard that argument before. Thats why at my company we have three incompatible, half arsed K8s clusters.

At the point where you have to fix upstream bugs, its the point where one says: fuckit, its not stable enough, more trouble than its worth. Lets use gaffer tape and move on. As for maintenance, without company buyin for transplanting the _entire_ stack, its questionable. And if there are only two people, and you have to maintain an entire distributed stack, that smacks of pain.

One company, one platform.

Re: Learning to operate Kubernetes reliably

#67

Post by SahAssar »

alexebird wrote: post #56

I always search for mentions of Hashicorp Nomad in the comments section of front-page Kubernetes articles like this. There are often few or no mentions, so I’d like to add a plug for the Hashistack. For some reason Nomad seems to get noticeably less publicity than some of the other Hashicorp offerings like Consul, Vault, and Terraform. In my opinion Nomad is right up there with them. The documentation is excellent. I…

What's your take on Fabio vs. Traefik? I had not heard of Fabio before, but they seem to support a similar featureset.

Re: Learning to operate Kubernetes reliably

#68

Post by obeattie »

alpb wrote: post #44
pishpash wrote: post #32

Earlier quoted context omitted.

I don't get this. Didn't Kubernetes come out of Google Borg that had been in use forever? The second write should be more elegant and impressive -- why so many basic bugs?

Kubernetes takes some concepts from Borg. A system like Borg would be very closely coupled to Google‘s infrastructure that there’s probably very little to open source from there without open sourcing the entire machinery. Also, any large scale system like Borg developed at a large company like Facebook or Google will have completely opinionated one-way-of-doing-things for a lot of aspects. This doesn’t work for the w…

I think this bit from "Borg, Omega, and Kubernetes"[1] (which is an excellent read) sheds light on this:

> The Borgmaster is a monolithic component that knows the semantics of every API operation. It contains the cluster management logic such as the state machines for jobs, tasks, and machines; and it runs the Paxos-based replicated storage system used to record the master’s state.

So it sounds as though Borg includes its own storage system. As I understand, Google has a set of (very complex) libraries written in C++ that implement Paxos/Multi-Paxos[2], which they have not open sourced.

[1] https://research.google.com/pubs/pub44843.html [2] https://research.google.com/archive/paxos_made_live.html

Re: Learning to operate Kubernetes reliably

#69

Post by toong »

alexebird wrote: post #56

I always search for mentions of Hashicorp Nomad in the comments section of front-page Kubernetes articles like this. There are often few or no mentions, so I’d like to add a plug for the Hashistack. For some reason Nomad seems to get noticeably less publicity than some of the other Hashicorp offerings like Consul, Vault, and Terraform. In my opinion Nomad is right up there with them. The documentation is excellent. I…

Catch 22: the lack of traction/adoption is the main point that stops me from exploring it more.

I would have to put so much effort in convincing customers and management to not go the (now almost default?) Kubernetes-route, that it's risky trying something else. A small hiccup in Nomad, would be enough for the pitchforks to come out.

Re: Learning to operate Kubernetes reliably

#70

Post by ownagefool »

KaiserPro wrote: post #66
ianstormtaylor wrote: post #36

Earlier quoted context omitted.

It feels like you didn’t read the article. The author made clear multiple times that they were using cron jobs as a test bed for Kubernetes, and they chose to “overengineer” because they’re looking to use Kubernetes for more and more of their needs over time. You’re kind of arguing against a straw man. I think it’s actually a great example of how Stripe thinks about technology choices. They’re interested in choosing…

I read the article, I understand completely and I've heard that argument before. Thats why at my company we have three incompatible, half arsed K8s clusters. At the point where you have to fix upstream bugs, its the point where one says: fuckit, its not stable enough, more trouble than its worth. Lets use gaffer tape and move on. As for maintenance, without company buyin for transplanting the _entire_ stack, its ques…

I think you're sort of right but not really.

If the benefits of running k8s outweigh the effort of kicking a few patches upstream. Further, if nobody is kicking patches upstream, where exactly are out open source solutions coming from?

I would counter argue about the times jenkins has bit me in the ass, but actually, most solutions will when you go deep enough.

Post reply on HN