Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1000 nodes and beyond: updates to Kubernetes performance and scalability

blog.kubernetes.io

Post reply on HN
Print view View original

31–40 of 59 posts

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#31

Post by justinsb »

Rapzid wrote: post #30
justinsb wrote: post #29

Earlier quoted context omitted.

1.2 includes multi-zone support, so your nodes can be in multiple AZs. This means that a failure of a single zone shouldn't interrupt your apps: http://kubernetes.io/docs/admin/multiple-zones/ What is not yet in 1.2, but is planned for 1.3, is HA Master - so that failure of the zone which contains your master won't interrupt the control plane. (i.e. you will be able to update your apps even as zones are failing).

Ah, nice! That wasn't super clear to me but now that you mentioned it, perhaps it should have been.

Not your fault - I was a little slow on getting the docs written up!

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#32

Post by Rapzid »

philips wrote: post #4
Rapzid wrote: post #3

1.2 has a lot of really nice additions such as infrastructure containers, the new config map API, service draining for node replacements, and many more. Unfortunately I would be running it on AWS and HA still hasn't been worked out and manual setup is a bear.

The community is working on what we like to call "self-hosted" Kubernetes. This will help reduce the complexity of installation on all platforms. You can see more about it from my KubeCon keynote: https://youtu.be/A49xXiKZNTQ?t=6m The target is to have this all upstream in the next (v1.3) release. Slides here: https://speakerdeck.com/philips/pushing-kubernetes-forward?s...

Awesome, will have a look. We run VPC per-environment, and support launching environments ad-hoc. So we would need to launch a Kubernetes cluster along with the environments. Anything to reduce the switching costs is very welcome :)

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#33

Post by bacheson1293 »

Dangeranger wrote: post #9

Does anyone have a good experience to share with a hosted Kubernetes provider outside of GCE and Tectonic? I am primarily comparing using Kubernetes to alternatives such as Rancher or Nomad.

Rancher now supports Kubernetes and Docker Swarm if you are not already aware.

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#34

Post by whalesalad »

We are running Kubernetes in production at FarmLogs and LOVE it. We're a very small team with a ton of operational work to do in other facets of the company as we prep for the season to begin, but once we've got some free time there will be an in-depth blog post describing our migration and roll-out. We've also built some really neat tooling that we would like to share with the world.

Upgrading to 1.2 has been incredible. Deployments are faster and pods get scheduled almost instantly now. Our Master nodes are down to about 1/4th of what they were normally doing in terms of CPU usage.

We're really excited to be ridin' on kubelets!

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#35

Post by xur17 »

whalesalad wrote: post #34

We are running Kubernetes in production at FarmLogs and LOVE it. We're a very small team with a ton of operational work to do in other facets of the company as we prep for the season to begin, but once we've got some free time there will be an in-depth blog post describing our migration and roll-out. We've also built some really neat tooling that we would like to share with the world. Upgrading to 1.2 has been incred…

I would be very interested to hear your process!

Our team is looking at using it, but we haven't found a great way to do automated deployments with our current build system (Bamboo). The best we've come up with is a series of bash scripts as the deployment step, but I'm not fully comfortable with how that would handle failed deployments yet. Basically, we need a way to handle automated deployments, and see the status of our currently deployed systems / promote environments.

If anyone is using Kubernetes in production, I'd love to hear what your deployment process looks like.

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#36

Post by knite »

philips wrote: post #6
Rapzid wrote: post #3

1.2 has a lot of really nice additions such as infrastructure containers, the new config map API, service draining for node replacements, and many more. Unfortunately I would be running it on AWS and HA still hasn't been worked out and manual setup is a bear.

kube-aws is a tool that we built at CoreOS to make installation of kubernetes on AWS easier. We just made a new release (v0.5.1) and would love feedback on that. It is what we use in production here at CoreOS. https://github.com/coreos/coreos-kubernetes/releases

kube-aws is great! When do you expect to update to Kubernetes 1.2?

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#37

Post by roberthbailey »

boulos wrote: post #8
rodionos wrote: post #7

The frame at 2:37 shows avg response time of 1.75 ms at 10 mln QPS. Which API call was measured? I'm looking at bar charts under "Metrics from Kubernetes 1.2" and the latencies graphed there appear to be different/higher.

That's the nginx response time. You can see that when he scales up the loadbots but not the backends and says that the "tail latency has gotten quite high" (about 1min in).

Correct. In addition, the source code used to run the demo is available on github at https://github.com/kubernetes/contrib/tree/master/scale-demo

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#39

Post by zurn »

boulos wrote: post #2

It's exciting to see that Kubernetes is ready for basically any scale. You're more likely to run out of quota (on your cloud provider, particularly IPs) or some other resource (on-prem) before you can't schedule a container quickly enough. Disclaimer: I work on Compute Engine and chat with the Kubernetes folks a lot.

It's hard to understand why so many cloud service providers and software stacks are still v4-only considering the operational and development cost of v4+NAT (complexity, management, scaling limitations etc). For most systems it would be enough for the front load balancers to speak v4.

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#40

Post by endymi0n »

zurn wrote: post #39
boulos wrote: post #2

It's exciting to see that Kubernetes is ready for basically any scale. You're more likely to run out of quota (on your cloud provider, particularly IPs) or some other resource (on-prem) before you can't schedule a container quickly enough. Disclaimer: I work on Compute Engine and chat with the Kubernetes folks a lot.

It's hard to understand why so many cloud service providers and software stacks are still v4-only considering the operational and development cost of v4+NAT (complexity, management, scaling limitations etc). For most systems it would be enough for the front load balancers to speak v4.

Funny, I see it exactly the other way around. I want NAT+Firewall to have at least decent perimeter security in a private LAN, the 10/8 subnet is large enough to do anything I can ever imagine to be doing and IPv4 is so much easier to grok. For most systems it would be enough for the front load balancers to speak v6.
Post reply on HN