Stateful Apps on Kubernetes: A quick primer
cockroachlabs.com
Stateful Apps on Kubernetes: A quick primer
1–10 of 36 posts
Re: Stateful Apps on Kubernetes: A quick primer
#2Has anyone looked at Service Fabric (Microsoft tech) for things like this? That has offered stateful services for years now. I'm pretty sure it runs on Linux, and I've seen that it's Docker compatible. I know it's kinda in the same space as K8s but I don't really know the details. Would SF be able to do something like this in a similar (or better?) way?
Re: Stateful Apps on Kubernetes: A quick primer
#3> Because Kubernetes itself runs on the machines that are running your databases, it will consume some resources and will slightly impact performance. In our testing, we found an approximately 5% dip in throughput on a simple key-value workload.
5% seems like a surprisingly large overhead. What is k8s doing in this situation that would have that kind of impact?
Re: Stateful Apps on Kubernetes: A quick primer
#4>Given its pedigree of literally working at Google-scale
I understood that a team at google developed k8s but google doesn't actually run it for their "google-scale" workloads. Am I misinformed?
Re: Stateful Apps on Kubernetes: A quick primer
#5>Given its pedigree of literally working at Google-scale I understood that a team at google developed k8s but google doesn't actually run it for their "google-scale" workloads. Am I misinformed?
You are correct.
> [kubernetes is] a simplified clone of Google’s internal borg system
https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48...
Re: Stateful Apps on Kubernetes: A quick primer
#6I would recommend against running stateful apps in kubernetes. It's not really ready for it. Big problems include routing (it works fine for http requests, but not for DBs, message brokers, etc) and just the pain of setting up stateful sets.
If you don't believe me, take it from someone who should know what they're talking about: https://twitter.com/kelseyhightower/status/96341350830081229...
Re: Stateful Apps on Kubernetes: A quick primer
#7>Given its pedigree of literally working at Google-scale I understood that a team at google developed k8s but google doesn't actually run it for their "google-scale" workloads. Am I misinformed?
You are correct. > [kubernetes is] a simplified clone of Google’s internal borg system https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48...
That said, Google, to my understanding, does run a completely containerized infrastructure internally, including databases and other stateful things, so it is not wildly off to suggest running a database on Kubernetes.
Re: Stateful Apps on Kubernetes: A quick primer
#8> Because Kubernetes itself runs on the machines that are running your databases, it will consume some resources and will slightly impact performance. In our testing, we found an approximately 5% dip in throughput on a simple key-value workload. 5% seems like a surprisingly large overhead. What is k8s doing in this situation that would have that kind of impact?
CPU Cache contention, network overhead introduced by Kubernetes service proxy model, even the liveness checks.
We haven’t yet evolved Kubernetes services to prefer specific cores and avoid app workloads quite yet (although cpu management is getting closer).
Docker is also somewhat hefty memory wise and you may contend on disk if not careful.
5% seems pretty reasonable to me in general, just as a consequence of having something heavier weight on the same node managing workloads.
Re: Stateful Apps on Kubernetes: A quick primer
#9> Because Kubernetes itself runs on the machines that are running your databases, it will consume some resources and will slightly impact performance. In our testing, we found an approximately 5% dip in throughput on a simple key-value workload. 5% seems like a surprisingly large overhead. What is k8s doing in this situation that would have that kind of impact?
Hard to say without knowing what they ran on what. There is a non trivial amount of memory that gets eaten up by the various k8s processes, docker and networking if you are using small nodes. I have a completely empty k8s cluster up right now with 1 worker and 1 master and the worker has about ~230 mb of ram used up.
Re: Stateful Apps on Kubernetes: A quick primer
#10Has anyone looked at Service Fabric (Microsoft tech) for things like this? That has offered stateful services for years now. I'm pretty sure it runs on Linux, and I've seen that it's Docker compatible. I know it's kinda in the same space as K8s but I don't really know the details. Would SF be able to do something like this in a similar (or better?) way?
It's complicated, because the definition of Service Fabric seems to be in flux.
The "original" Service Fabric is a high-level framework which requires invasive source code changes (you can't just drop an existing app on top of it), but gives you lots of benefits (scale, reliability etc) if you make the effort.
Recently container-based platforms - Docker, Kubernetes, etc - have come along with a different tradeoff: better compatibility with existing applications in exchange for less magical benefits. That approach is getting much more traction, and I think internally at Microsoft there is some infighting between the "Service Fabric camp" and the "Containers camp". One consequence of the infighting is that Service Fabric is extending its scope to include features like "container support". It's not clear to what extent that is done in collaboration with the "container people", or as a way to bypass them. I think they are still trying to decide whether to embrace Kubernetes, or replicate the functionality in-house. My prediction is that the container-based approach will win, but if will take time for the politics to fully play out. In the meantime things will continue to be confusing.
Bottom line: when evaluating Service Fabric, watch out for confusing and inconsistent use of the brand. It's a common pattern with large vendors - for example IBM with "Bluemix", SAP with "Hana", etc.