Curious to know why they are choosing to go with protobuf for intracluster communications as opposed to zero copy protocols like capn proto or flatbuffers. No doubt protobufs are probably much more battle tested in google scale environments, but are there any other clear benefits if the goal is to reduce spending cpu time encoding/decoding messages? Especially in SOA deployments where many small services need to comm…
1000 nodes and beyond: updates to Kubernetes performance and scalability
51–59 of 59 posts
Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability
#52Earlier quoted context omitted.
It's hard to understand why so many cloud service providers and software stacks are still v4-only considering the operational and development cost of v4+NAT (complexity, management, scaling limitations etc). For most systems it would be enough for the front load balancers to speak v4.
Funny, I see it exactly the other way around. I want NAT+Firewall to have at least decent perimeter security in a private LAN, the 10/8 subnet is large enough to do anything I can ever imagine to be doing and IPv4 is so much easier to grok. For most systems it would be enough for the front load balancers to speak v6.
Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability
#53Earlier quoted context omitted.
That's the nginx response time. You can see that when he scales up the loadbots but not the backends and says that the "tail latency has gotten quite high" (about 1min in).
Correct. In addition, the source code used to run the demo is available on github at https://github.com/kubernetes/contrib/tree/master/scale-demo
Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability
#54And still no way to make a simple 2-node cluster in 2 different availability zones. What if one AZ fails completely? Happens quite often. I tried to read HA documentation on Kubernetes and it all starts with warnings like "this is fairly advanced stuff, requiring intimate knowledge of Kubernetes inner workings", and going on with pages and pages of setup process. Basic HA is not a "fairly advanced stuff", it is a com…
I still wonder what is the primary use case for Kubernetes (or Docker Swarm, which has similar issues) if high availability is so low on the priority list.
Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability
#55And still no way to make a simple 2-node cluster in 2 different availability zones. What if one AZ fails completely? Happens quite often. I tried to read HA documentation on Kubernetes and it all starts with warnings like "this is fairly advanced stuff, requiring intimate knowledge of Kubernetes inner workings", and going on with pages and pages of setup process. Basic HA is not a "fairly advanced stuff", it is a com…
Conceptually speaking, having two nodes is not high availability, it is failover / fault tolerance. High availability is generally N + 2 where N is >= 1. This is a better explanation than I would write on this: https://www.quora.com/What-is-the-difference-between-a-highl...
Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability
#56Earlier quoted context omitted.
I would be very interested to hear your process! Our team is looking at using it, but we haven't found a great way to do automated deployments with our current build system (Bamboo). The best we've come up with is a series of bash scripts as the deployment step, but I'm not fully comfortable with how that would handle failed deployments yet. Basically, we need a way to handle automated deployments, and see the status…
We're looking at Spinnaker.io + Kubernetes right now for just this reason. K8s support was just added.
Please feel free to email me (aronchick (at) google) if you'd like to discuss this further (either P or GP post). We've seen a lot of this, and would love to help you out!
Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability
#57Earlier quoted context omitted.
We're looking at Spinnaker.io + Kubernetes right now for just this reason. K8s support was just added.
Disclosure: I work at Google on Kubernetes. Please feel free to email me (aronchick (at) google) if you'd like to discuss this further (either P or GP post). We've seen a lot of this, and would love to help you out!
Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability
#58Earlier quoted context omitted.
Conceptually speaking, having two nodes is not high availability, it is failover / fault tolerance. High availability is generally N + 2 where N is >= 1. This is a better explanation than I would write on this: https://www.quora.com/What-is-the-difference-between-a-highl...
You are right, of course. Still, I don't understand why it is so low priority in container orchestration platforms. And how it is even possible to live without it in production.
Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability
#59We are running Kubernetes in production at FarmLogs and LOVE it. We're a very small team with a ton of operational work to do in other facets of the company as we prep for the season to begin, but once we've got some free time there will be an in-depth blog post describing our migration and roll-out. We've also built some really neat tooling that we would like to share with the world. Upgrading to 1.2 has been incred…
I would be very interested to hear your process! Our team is looking at using it, but we haven't found a great way to do automated deployments with our current build system (Bamboo). The best we've come up with is a series of bash scripts as the deployment step, but I'm not fully comfortable with how that would handle failed deployments yet. Basically, we need a way to handle automated deployments, and see the status…