Live data from Hacker News

1000 nodes and beyond: updates to Kubernetes performance and scalability

blog.kubernetes.io

51–59 of 59 posts

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#51

Curious to know why they are choosing to go with protobuf for intracluster communications as opposed to zero copy protocols like capn proto or flatbuffers. No doubt protobufs are probably much more battle tested in google scale environments, but are there any other clear benefits if the goal is to reduce spending cpu time encoding/decoding messages? Especially in SOA deployments where many small services need to comm…

Gory details: https://github.com/kubernetes/kubernetes/issues/8132

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#52
post #39

Earlier quoted context omitted.

It's hard to understand why so many cloud service providers and software stacks are still v4-only considering the operational and development cost of v4+NAT (complexity, management, scaling limitations etc). For most systems it would be enough for the front load balancers to speak v4.

Funny, I see it exactly the other way around. I want NAT+Firewall to have at least decent perimeter security in a private LAN, the 10/8 subnet is large enough to do anything I can ever imagine to be doing and IPv4 is so much easier to grok. For most systems it would be enough for the front load balancers to speak v6.

Then you just have a system that has no security advantages but is harder to reason about due to the additional level of indirection in overloaded addressing (because of ambiguous addressing, management of forwarding rules, etc) vs normal firewalling without NAT. Which equates to some loss of security on the system level, because you can only effectively secure systems you understand.

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#53
post #8

Earlier quoted context omitted.

That's the nginx response time. You can see that when he scales up the loadbots but not the backends and says that the "tail latency has gotten quite high" (about 1min in).

Correct. In addition, the source code used to run the demo is available on github at https://github.com/kubernetes/contrib/tree/master/scale-demo

Thanks, all clear now.

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#54

And still no way to make a simple 2-node cluster in 2 different availability zones. What if one AZ fails completely? Happens quite often. I tried to read HA documentation on Kubernetes and it all starts with warnings like "this is fairly advanced stuff, requiring intimate knowledge of Kubernetes inner workings", and going on with pages and pages of setup process. Basic HA is not a "fairly advanced stuff", it is a com…

Update2: persistent volumes are still allocated only in the same AZ with master container. Hence, no HA databases (only manual volume provisioning is possible).

I still wonder what is the primary use case for Kubernetes (or Docker Swarm, which has similar issues) if high availability is so low on the priority list.

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#55
post #50

And still no way to make a simple 2-node cluster in 2 different availability zones. What if one AZ fails completely? Happens quite often. I tried to read HA documentation on Kubernetes and it all starts with warnings like "this is fairly advanced stuff, requiring intimate knowledge of Kubernetes inner workings", and going on with pages and pages of setup process. Basic HA is not a "fairly advanced stuff", it is a com…

Conceptually speaking, having two nodes is not high availability, it is failover / fault tolerance. High availability is generally N + 2 where N is >= 1. This is a better explanation than I would write on this: https://www.quora.com/What-is-the-difference-between-a-highl...

You are right, of course. Still, I don't understand why it is so low priority in container orchestration platforms. And how it is even possible to live without it in production.

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#56
post #35

Earlier quoted context omitted.

I would be very interested to hear your process! Our team is looking at using it, but we haven't found a great way to do automated deployments with our current build system (Bamboo). The best we've come up with is a series of bash scripts as the deployment step, but I'm not fully comfortable with how that would handle failed deployments yet. Basically, we need a way to handle automated deployments, and see the status…

We're looking at Spinnaker.io + Kubernetes right now for just this reason. K8s support was just added.

Disclosure: I work at Google on Kubernetes.

Please feel free to email me (aronchick (at) google) if you'd like to discuss this further (either P or GP post). We've seen a lot of this, and would love to help you out!

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#57

Earlier quoted context omitted.

We're looking at Spinnaker.io + Kubernetes right now for just this reason. K8s support was just added.

Disclosure: I work at Google on Kubernetes. Please feel free to email me (aronchick (at) google) if you'd like to discuss this further (either P or GP post). We've seen a lot of this, and would love to help you out!

Awesome, thanks for the offer! I'll reach out in the next day or two! :)

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#58
post #50

Earlier quoted context omitted.

Conceptually speaking, having two nodes is not high availability, it is failover / fault tolerance. High availability is generally N + 2 where N is >= 1. This is a better explanation than I would write on this: https://www.quora.com/What-is-the-difference-between-a-highl...

You are right, of course. Still, I don't understand why it is so low priority in container orchestration platforms. And how it is even possible to live without it in production.

It's not low on the priority list at all. These are the same people who worked on borg (I'm a contributor, but didn't work on borg); they get stateful applications and understand that it needs to be done RIGHT. No second chances. Nailing this for 1.0 or 1.1 would have consumed a significant portion of the team, but rest assured it will work, soon.

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#59
post #35

We are running Kubernetes in production at FarmLogs and LOVE it. We're a very small team with a ton of operational work to do in other facets of the company as we prep for the season to begin, but once we've got some free time there will be an in-depth blog post describing our migration and roll-out. We've also built some really neat tooling that we would like to share with the world. Upgrading to 1.2 has been incred…

I would be very interested to hear your process! Our team is looking at using it, but we haven't found a great way to do automated deployments with our current build system (Bamboo). The best we've come up with is a series of bash scripts as the deployment step, but I'm not fully comfortable with how that would handle failed deployments yet. Basically, we need a way to handle automated deployments, and see the status…

For GitLab CI we just released deploy to Kubernetes https://about.gitlab.com/2016/03/22/gitlab-8-6-released/
Post reply on HN