Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1000 nodes and beyond: updates to Kubernetes performance and scalability

blog.kubernetes.io

Post reply on HN
Print view View original

41–50 of 59 posts

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#41

Post by deepanchor »

Curious to know why they are choosing to go with protobuf for intracluster communications as opposed to zero copy protocols like capn proto or flatbuffers.

No doubt protobufs are probably much more battle tested in google scale environments, but are there any other clear benefits if the goal is to reduce spending cpu time encoding/decoding messages?

Especially in SOA deployments where many small services need to communicate with one another, I would think that the ability to quickly read any field from a message and pass it on (without first having to decode the entire message) would be a very desirable trait.

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#42

Post by meddlepal »

xur17 wrote: post #35
whalesalad wrote: post #34

We are running Kubernetes in production at FarmLogs and LOVE it. We're a very small team with a ton of operational work to do in other facets of the company as we prep for the season to begin, but once we've got some free time there will be an in-depth blog post describing our migration and roll-out. We've also built some really neat tooling that we would like to share with the world. Upgrading to 1.2 has been incred…

I would be very interested to hear your process! Our team is looking at using it, but we haven't found a great way to do automated deployments with our current build system (Bamboo). The best we've come up with is a series of bash scripts as the deployment step, but I'm not fully comfortable with how that would handle failed deployments yet. Basically, we need a way to handle automated deployments, and see the status…

We're looking at Spinnaker.io + Kubernetes right now for just this reason. K8s support was just added.

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#43

Post by vetinari »

TheIronYuppie wrote: post #22
tzaman wrote: post #21

Earlier quoted context omitted.

That resource being money. I'm deploying a fairly simple app on GKE and things go out of hand quickly due to confusing pricing. Or maybe I just don't where to look.

Disclaimer: I work at Google on Kubernetes. Can you say more? Did you just spin up too many nodes?

I would say that there's impedance mismatch between GKE pricing and unclear requirements, how much resources in what structure you will need.

I was looking at the Kubernetes tutorials and couldn't even start to figure out, how much would it cost to run them. (Well, I didn't try too hard, it wasn't that important.)

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#44

Post by kentonv »

deepanchor wrote: post #41

Curious to know why they are choosing to go with protobuf for intracluster communications as opposed to zero copy protocols like capn proto or flatbuffers. No doubt protobufs are probably much more battle tested in google scale environments, but are there any other clear benefits if the goal is to reduce spending cpu time encoding/decoding messages? Especially in SOA deployments where many small services need to comm…

Protobuf is the Google standard, used by basically every single server at Google for the last 15 years. They have built an internal ecosystem of tools around the format. For a Google project to use something different would be weird and would face lots of internal push-back, for good reasons.

Even though FlatBuffers is technically from Google, it's from a sub-team of Android working on tools aimed at Android games. The idea was that you'd store your assets in this format. IIRC the initial release didn't do bounds checking so was totally vulnerable to malicious input (but it wasn't intended for such use cases anyhow). I doubt it is widely used on Google's servers.

Cap'n Proto is not from Google and there's simply no way they'd choose to use it. To be fair, its support for languages other than C++ remains weak, largely because Sandstorm.io doesn't currently have the resources to build it out.

FWIW the ability to read a single field from a message is less important in networking situations because sending/receiving the message is already O(n) and the messages are small-ish, so parsing in O(n) is not a huge deal. Random-access parsing really shines when the input is a massive file on disk.

(I'm the author of Cap'n Proto and also of Protobuf v2 (the first version Google open sourced).)

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#46

Post by atemerev »

And still no way to make a simple 2-node cluster in 2 different availability zones. What if one AZ fails completely? Happens quite often.

I tried to read HA documentation on Kubernetes and it all starts with warnings like "this is fairly advanced stuff, requiring intimate knowledge of Kubernetes inner workings", and going on with pages and pages of setup process.

Basic HA is not a "fairly advanced stuff", it is a commonplace requirement in any production environment. Why do I need a 1000-node cluster if all 1000 nodes are in the same AZ, which can have an outage anytime?

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#47

Post by atemerev »

justinsb wrote: post #29
Rapzid wrote: post #3

1.2 has a lot of really nice additions such as infrastructure containers, the new config map API, service draining for node replacements, and many more. Unfortunately I would be running it on AWS and HA still hasn't been worked out and manual setup is a bear.

1.2 includes multi-zone support, so your nodes can be in multiple AZs. This means that a failure of a single zone shouldn't interrupt your apps: http://kubernetes.io/docs/admin/multiple-zones/ What is not yet in 1.2, but is planned for 1.3, is HA Master - so that failure of the zone which contains your master won't interrupt the control plane. (i.e. you will be able to update your apps even as zones are failing).

Oh, cool. That's what I was looking for in the docs for the last 2 days.

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#48

Post by atemerev »

atemerev wrote: post #46

And still no way to make a simple 2-node cluster in 2 different availability zones. What if one AZ fails completely? Happens quite often. I tried to read HA documentation on Kubernetes and it all starts with warnings like "this is fairly advanced stuff, requiring intimate knowledge of Kubernetes inner workings", and going on with pages and pages of setup process. Basic HA is not a "fairly advanced stuff", it is a com…

Update: docs just arrived — http://kubernetes.io/docs/admin/multiple-zones/ . That's better :)

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#49

Post by kiallmacinnes »

endymi0n wrote: post #40
zurn wrote: post #39

Earlier quoted context omitted.

It's hard to understand why so many cloud service providers and software stacks are still v4-only considering the operational and development cost of v4+NAT (complexity, management, scaling limitations etc). For most systems it would be enough for the front load balancers to speak v4.

Funny, I see it exactly the other way around. I want NAT+Firewall to have at least decent perimeter security in a private LAN, the 10/8 subnet is large enough to do anything I can ever imagine to be doing and IPv4 is so much easier to grok. For most systems it would be enough for the front load balancers to speak v6.

Wanting a firewall is good, you can have that on v4 or v6. However, after you add 3 simple firewall rules, NAT provides no additional security.

   Drop state=INVALID
   Allow state=EATABLISHED,RELATED
   Drop all
You're now just as secure as if you had a typical NAT setup, but without the decades of kludges that is NAT.

Re: 1000 nodes and beyond: updates to Kubernetes performance and scalability

#50

Post by SEJeff »

atemerev wrote: post #46

And still no way to make a simple 2-node cluster in 2 different availability zones. What if one AZ fails completely? Happens quite often. I tried to read HA documentation on Kubernetes and it all starts with warnings like "this is fairly advanced stuff, requiring intimate knowledge of Kubernetes inner workings", and going on with pages and pages of setup process. Basic HA is not a "fairly advanced stuff", it is a com…

Conceptually speaking, having two nodes is not high availability, it is failover / fault tolerance. High availability is generally N + 2 where N is >= 1.

This is a better explanation than I would write on this:

https://www.quora.com/What-is-the-difference-between-a-highl...

Post reply on HN