Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stateful Apps on Kubernetes: A quick primer

cockroachlabs.com

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Stateful Apps on Kubernetes: A quick primer

#21

Post by puzzle »

outside1234 wrote: post #7
bajsejohannes wrote: post #5

Earlier quoted context omitted.

You are correct. > [kubernetes is] a simplified clone of Google’s internal borg system https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48...

That said, Google, to my understanding, does run a completely containerized infrastructure internally, including databases and other stateful things, so it is not wildly off to suggest running a database on Kubernetes.

Before it ran on F1/Spanner, Adwords ran on sharded MySQL on Borg. Not for its entire early life, but for quite a few years. Later in life, Checkout and maybe Wallet ran on MySQL on Borg, too. So did YouTube, which used Vitess, a sharding layer (now ported to Kubernetes and open sourced).

Bigtable, Spanner and even Colossus/D run in containers on Borg.

Re: Stateful Apps on Kubernetes: A quick primer

#22

Post by puzzle »

kyrra wrote: post #19
outside1234 wrote: post #7

Earlier quoted context omitted.

That said, Google, to my understanding, does run a completely containerized infrastructure internally, including databases and other stateful things, so it is not wildly off to suggest running a database on Kubernetes.

Packaging was (is?) a bit different, because it existed before Docker existed. Pdf of the presentation: https://www.usenix.org/sites/default/files/conference/protec... And a recording of the talk: https://www.usenix.org/conference/lisa14/conference-program/...

Googlers, including the ones working on Kubernetes, like to pick on MPM, but that's probably because they haven't run major services deployed from Docker images.

Re: Stateful Apps on Kubernetes: A quick primer

#23

Post by puzzle »

bajsejohannes wrote: post #5
lowbloodsugar wrote: post #4

>Given its pedigree of literally working at Google-scale I understood that a team at google developed k8s but google doesn't actually run it for their "google-scale" workloads. Am I misinformed?

You are correct. > [kubernetes is] a simplified clone of Google’s internal borg system https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48...

Kubernetes might be simpler than Borg in so many ways (let me count the ones I care about...), but it also has better features that Borg did not implement (labels and selectors) or that are offered only by some other internal services, which obviously are configured through very different mechanisms (ingress).

Re: Stateful Apps on Kubernetes: A quick primer

#25

Post by tayo42 »

puzzle wrote: post #21
outside1234 wrote: post #7

Earlier quoted context omitted.

That said, Google, to my understanding, does run a completely containerized infrastructure internally, including databases and other stateful things, so it is not wildly off to suggest running a database on Kubernetes.

Before it ran on F1/Spanner, Adwords ran on sharded MySQL on Borg. Not for its entire early life, but for quite a few years. Later in life, Checkout and maybe Wallet ran on MySQL on Borg, too. So did YouTube, which used Vitess, a sharding layer (now ported to Kubernetes and open sourced). Bigtable, Spanner and even Colossus/D run in containers on Borg.

How is colossus running in a container in borg? That's their network file system that makes running databases in containers possible?

Re: Stateful Apps on Kubernetes: A quick primer

#26

Post by puzzle »

tayo42 wrote: post #25
puzzle wrote: post #21

Earlier quoted context omitted.

Before it ran on F1/Spanner, Adwords ran on sharded MySQL on Borg. Not for its entire early life, but for quite a few years. Later in life, Checkout and maybe Wallet ran on MySQL on Borg, too. So did YouTube, which used Vitess, a sharding layer (now ported to Kubernetes and open sourced). Bigtable, Spanner and even Colossus/D run in containers on Borg.

How is colossus running in a container in borg? That's their network file system that makes running databases in containers possible?

There's a bootstrapping issue, of course, but that's how it works.

And it's even crazier than what you're picturing. What if I told you that Bigtable runs on top of Colossus, but Colossus itself stores metadata about files, including Bigtable's files, in... Bigtable? It's really turtles all the way down, the last of which is, luckily, Chubby.

Some of the details here: http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke...

Re: Stateful Apps on Kubernetes: A quick primer

#27

Post by AndyNemmity »

bajsejohannes wrote: post #6

I would recommend against running stateful apps in kubernetes. It's not really ready for it. Big problems include routing (it works fine for http requests, but not for DBs, message brokers, etc) and just the pain of setting up stateful sets. If you don't believe me, take it from someone who should know what they're talking about: https://twitter.com/kelseyhightower/status/96341350830081229...

We run many, many stateful apps on kubernetes. Not without challenges certainly, but I am not sure any of them are really kubernetes specific.

They just don't act like other services, and require more care. That's about it. I think that's what Kelsey is referring to, you can't just treat them the same as other pods.

Re: Stateful Apps on Kubernetes: A quick primer

#28

Post by a-robinson »

smarterclayton wrote: post #8
rrdharan wrote: post #3

> Because Kubernetes itself runs on the machines that are running your databases, it will consume some resources and will slightly impact performance. In our testing, we found an approximately 5% dip in throughput on a simple key-value workload. 5% seems like a surprisingly large overhead. What is k8s doing in this situation that would have that kind of impact?

CPU Cache contention, network overhead introduced by Kubernetes service proxy model, even the liveness checks. We haven’t yet evolved Kubernetes services to prefer specific cores and avoid app workloads quite yet (although cpu management is getting closer). Docker is also somewhat hefty memory wise and you may contend on disk if not careful. 5% seems pretty reasonable to me in general, just as a consequence of having…

Yeah, it appeared to just be general resource contention from having to share the machine -- CPU interrupts, less memory available, etc.

I'll note, though, that the 5% number is when using host networking for both Cockroach and the client load generator. Using GKE's default cluster networking through the Docker bridge is closer to 15% worse than running directly on equivalent non-Kubernetes VMs.

Re: Stateful Apps on Kubernetes: A quick primer

#29

Post by manigandham »

daxfohl wrote: post #12
bajsejohannes wrote: post #5

Earlier quoted context omitted.

You are correct. > [kubernetes is] a simplified clone of Google’s internal borg system https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48...

Wow, I'd never heard of Fargate until this. It seems so somewhat obvious now for things that Lambda doesn't fit. Heroku for containers, not VMs.

There's also Azure Container Instances: https://azure.microsoft.com/en-us/services/container-instanc...

And Google has AppEngine, now with Flexible Environment that can run containers: https://cloud.google.com/appengine/docs/flexible/dotnet/quic...

Re: Stateful Apps on Kubernetes: A quick primer

#30

Post by merb »

hardwaresofton wrote: post #17
loiselleatwork wrote: post #14

Earlier quoted context omitted.

Author of post here and CRL employee: just for some additional detail, we reached out to Kelsey about the problems he's seen running databases in Kubernetes. https://twitter.com/kelseyhightower/status/96347131657256140... He said "You still need to worry about database backups and restores. You need to consider downtime during cluster upgrades." These things are totally true. K8s doesn't automate backups (edit: by de…

Note that none of those things are impossible on kubernetes, k8s just doesn't offer them by default (which is good IMO). There are projects that help you run databases in kubernetes and also make backups of many things hosted: - Automatic CephFS for your cluster -> https://rook.io/docs/rook/master/ - Backups for cluster resources and volumes -> https://github.com/heptio/ark - Spin up dynamic postgres clusters -> http…

Well with k8s 1.10+ it's also possible to use statefulsets and local volume, so with affinity it's possible to just use k8s as an orchestration system where you "install" your database and keep it up to date with k8s. of course if a node goes down you need to failover, etc. but patroni/zalando postgres works really well with statefulsets and local volume. (as long as a single node is still running, which should always be the case...) (https://kubernetes.io/blog/2018/04/13/local-persistent-volum...)
Post reply on HN