Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stateful Apps on Kubernetes: A quick primer

cockroachlabs.com

Post reply on HN
Print view View original

11–20 of 36 posts

Re: Stateful Apps on Kubernetes: A quick primer

#11

Post by daxfohl »

zapita wrote: post #10
daxfohl wrote: post #2

Has anyone looked at Service Fabric (Microsoft tech) for things like this? That has offered stateful services for years now. I'm pretty sure it runs on Linux, and I've seen that it's Docker compatible. I know it's kinda in the same space as K8s but I don't really know the details. Would SF be able to do something like this in a similar (or better?) way?

It's complicated, because the definition of Service Fabric seems to be in flux. The "original" Service Fabric is a high-level framework which requires invasive source code changes (you can't just drop an existing app on top of it), but gives you lots of benefits (scale, reliability etc) if you make the effort. Recently container-based platforms - Docker, Kubernetes, etc - have come along with a different tradeoff: be…

Okay that's about what it looked like to me too. There's only so many magic words you can throw at a tech and expect it to work together happily. Looking into it, the stateful service side of SF doesn't seem particularly compatible with the container side of it. A stateful service is a stateful SF service, and a container service is its own thing. Maybe there's a way to plug them together but unfortunately I didn't see it.

Re: Stateful Apps on Kubernetes: A quick primer

#12

Post by daxfohl »

bajsejohannes wrote: post #5
lowbloodsugar wrote: post #4

>Given its pedigree of literally working at Google-scale I understood that a team at google developed k8s but google doesn't actually run it for their "google-scale" workloads. Am I misinformed?

You are correct. > [kubernetes is] a simplified clone of Google’s internal borg system https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48...

Wow, I'd never heard of Fargate until this. It seems so somewhat obvious now for things that Lambda doesn't fit. Heroku for containers, not VMs.

Re: Stateful Apps on Kubernetes: A quick primer

#13

Post by williamstein »

bajsejohannes wrote: post #6

I would recommend against running stateful apps in kubernetes. It's not really ready for it. Big problems include routing (it works fine for http requests, but not for DBs, message brokers, etc) and just the pain of setting up stateful sets. If you don't believe me, take it from someone who should know what they're talking about: https://twitter.com/kelseyhightower/status/96341350830081229...

I've used Kubernetes a massive amount during the last two years for running stateful apps. In contrast, I do recommend it. Yes, it is challenging, since stateful app are. However, the challenges are all well worth solving in the context of Kubernetes (great benefits from health checks, automated reproducible deployments, etc.). The situation is pretty good these days in my experience; at least, a lot better than 2 years ago!

Re: Stateful Apps on Kubernetes: A quick primer

#14

Post by loiselleatwork »

bajsejohannes wrote: post #6

I would recommend against running stateful apps in kubernetes. It's not really ready for it. Big problems include routing (it works fine for http requests, but not for DBs, message brokers, etc) and just the pain of setting up stateful sets. If you don't believe me, take it from someone who should know what they're talking about: https://twitter.com/kelseyhightower/status/96341350830081229...

Author of post here and CRL employee: just for some additional detail, we reached out to Kelsey about the problems he's seen running databases in Kubernetes.

https://twitter.com/kelseyhightower/status/96347131657256140...

He said "You still need to worry about database backups and restores. You need to consider downtime during cluster upgrades."

These things are totally true. K8s doesn't automate backups (edit: by default; though, it can) and if you need to take K8s down for upgrades, then everything is down. For its part, though, CockroachDB supports rolling upgrades with no downtime on Kubernetes.

As for routing, that is tough problem if you want to run K8s across multiple regions, though we have some folks who've done it.

And if one finds setting up StatefulSets challenging, we have a tutorial on how to do it written by a former Kubernetes engineer: https://www.cockroachlabs.com/docs/stable/orchestrate-cockro...

Re: Stateful Apps on Kubernetes: A quick primer

#15

Post by bajsejohannes »

williamstein wrote: post #13
bajsejohannes wrote: post #6

I would recommend against running stateful apps in kubernetes. It's not really ready for it. Big problems include routing (it works fine for http requests, but not for DBs, message brokers, etc) and just the pain of setting up stateful sets. If you don't believe me, take it from someone who should know what they're talking about: https://twitter.com/kelseyhightower/status/96341350830081229...

I've used Kubernetes a massive amount during the last two years for running stateful apps. In contrast, I do recommend it. Yes, it is challenging, since stateful app are. However, the challenges are all well worth solving in the context of Kubernetes (great benefits from health checks, automated reproducible deployments, etc.). The situation is pretty good these days in my experience; at least, a lot better than 2 ye…

Good point about it getting better. A lot of the pain was from trying to do it before 1.8.

Re: Stateful Apps on Kubernetes: A quick primer

#16

Post by lobster_johnson »

bajsejohannes wrote: post #6

I would recommend against running stateful apps in kubernetes. It's not really ready for it. Big problems include routing (it works fine for http requests, but not for DBs, message brokers, etc) and just the pain of setting up stateful sets. If you don't believe me, take it from someone who should know what they're talking about: https://twitter.com/kelseyhightower/status/96341350830081229...

We run stateful apps on Kubernetes. There are obvious rough areas (the lack of persistent volume resizing, for example, which is scheduled for 1.11), but overall, it's great.

What a lot of naysayers leave out, or choose to ignore, is that the challenges running stateful apps on Kubernetes mirror those of running stateful apps anywhere. If you run Postgres on a VM, for example, you're completely reliant on that VM staying up -- this is no different from Kubernetes. Some will also point out the dangers of co-locating lots of software (such as Postgres) on the same machine as many other containers, as they will compete for CPU and I/O; but this is also no different than on Kubernetes, which provides plenty of tools (affinities/anti-affinities, node selectors) to isolate containers to machines. And so on. Containers bring some new challenges, but Kubernetes meets them quite well.

What specific issues do you have? I'm not sure I understand the point about routing. I also don't understand what the "pain" of stateful sets refers to.

Re: Stateful Apps on Kubernetes: A quick primer

#17

Post by hardwaresofton »

loiselleatwork wrote: post #14
bajsejohannes wrote: post #6

I would recommend against running stateful apps in kubernetes. It's not really ready for it. Big problems include routing (it works fine for http requests, but not for DBs, message brokers, etc) and just the pain of setting up stateful sets. If you don't believe me, take it from someone who should know what they're talking about: https://twitter.com/kelseyhightower/status/96341350830081229...

Author of post here and CRL employee: just for some additional detail, we reached out to Kelsey about the problems he's seen running databases in Kubernetes. https://twitter.com/kelseyhightower/status/96347131657256140... He said "You still need to worry about database backups and restores. You need to consider downtime during cluster upgrades." These things are totally true. K8s doesn't automate backups (edit: by de…

Note that none of those things are impossible on kubernetes, k8s just doesn't offer them by default (which is good IMO).

There are projects that help you run databases in kubernetes and also make backups of many things hosted:

- Automatic CephFS for your cluster -> https://rook.io/docs/rook/master/

- Backups for cluster resources and volumes -> https://github.com/heptio/ark

- Spin up dynamic postgres clusters -> https://github.com/zalando-incubator/postgres-operator

Databases are just applications with different resource needs. Please stop pushing forward the notion that they can't be run in containers or container orchestration systems. Databases are just programs. If the substrate for running your containers doensn't reliably support flock or fsync or something your database needs, then maybe pick a better substrate that does -- container runtimes these days and kubernetes don't stand in your way these days.

Re: Stateful Apps on Kubernetes: A quick primer

#19

Post by kyrra »

outside1234 wrote: post #7
bajsejohannes wrote: post #5

Earlier quoted context omitted.

You are correct. > [kubernetes is] a simplified clone of Google’s internal borg system https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48...

That said, Google, to my understanding, does run a completely containerized infrastructure internally, including databases and other stateful things, so it is not wildly off to suggest running a database on Kubernetes.

Packaging was (is?) a bit different, because it existed before Docker existed.

Pdf of the presentation: https://www.usenix.org/sites/default/files/conference/protec...

And a recording of the talk: https://www.usenix.org/conference/lisa14/conference-program/...

Re: Stateful Apps on Kubernetes: A quick primer

#20

Post by lobster_johnson »

lowbloodsugar wrote: post #4

>Given its pedigree of literally working at Google-scale I understood that a team at google developed k8s but google doesn't actually run it for their "google-scale" workloads. Am I misinformed?

It's my understanding, based on comments by googlers here on HN, that Google does run a bunch of apps on GKE. We don't know about which apps, but it's not surprising that they want to dogfood their own cloud platform.
Post reply on HN