Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Automatic K8s pod placement to match external service zones

github.com

Post reply on HN
Print view View original

11–20 of 47 posts

Re: Automatic K8s pod placement to match external service zones

#11

Post by toredash »

dserodio wrote: post #7
toredash wrote: post #6

Earlier quoted context omitted.

I'm not sure I follow. Are you talking about the AZP service, or ... ?

It's a best practice to have a Deployment run multiple Pods in separate AZs to increase availability

Yes I get that. But are we talking HA for this lookup service that I've made?

If yes, that's a simple update of the manifest to have 3 replicas with ab affinity setting to spread that out over different AZ. Kyverno would use the internal Service object this service provide to have a HA endpoint to send queries to.

If we are not talking about this AZP service, I don't understand what we are talkin about.

Re: Automatic K8s pod placement to match external service zones

#12

Post by mystifyingpoi »

ruuda wrote: post #8

> Have you considered alternative solutions? How about, don't use Kubernetes? The lack of control over where the workload runs is a problem caused by Kubernetes. If you deploy an application as e.g. systemd services, you can pick the optimal host for the workload, and it will not suddenly jump around.

> The lack of control

This project literally sets the affinity. That's precisely the control you seem to negate.

Re: Automatic K8s pod placement to match external service zones

#13

Post by stackskipton »

toredash wrote: post #2

Hi HN, I wanted to share something I've worked a bit to solve regarding Kubernetes: its scheduler has no awareness of the network topology for external services that workloads communicate with. If a pod talks to a database (e.g AWS RDS), K8s does not know it should schedule it in the same AZ as the database. If placed in the wrong AZ, it leads to unnecessary cross-AZ network traffic, adding latency (and costs $). I'v…

How do you handle RDS failovers? Mutating Webhook is only fired when Pods are created so if AZ zone does not fail, there is no pods to be created and affinity rules to be changed.

Re: Automatic K8s pod placement to match external service zones

#14

Post by stackskipton »

This is one of those ideas that sounds great and appears simple at first but can grow into mad monster. Here my potential thoughts after 5 minutes.

Kyverno requirement makes it limited. There is no "automatic-zone-placement-disabled" function in case someone wants to temporarily disable zone placement but not remove the label. How do we handle RDS Zone changing after workload scheduling? No automatic look up of IPs and Zones. What if we only have one node in specific zone? Are we willing to handle EC2 failure or should we trigger scale out?

Re: Automatic K8s pod placement to match external service zones

#15

Post by toredash »

ruuda wrote: post #8

> Have you considered alternative solutions? How about, don't use Kubernetes? The lack of control over where the workload runs is a problem caused by Kubernetes. If you deploy an application as e.g. systemd services, you can pick the optimal host for the workload, and it will not suddenly jump around.

Agree, Kubernetes isn't for everyone. This solution came from an specific issue with a client which had ad hoc performance problems when a Pod was placed in the "in-correct" AZ. So this solution was created to place the Pods in the most optimal zone when they were created.

Re: Automatic K8s pod placement to match external service zones

#16

Post by toredash »

mystifyingpoi wrote: post #10

Cool idea, I like that. Though I'm curious about the lookup service. You say: > To gather zone information, use this command ... Why couldn't most of this information be gathered by lookup service itself? A point could be made about excessive IAM, but a simple case of RDS reader residing in a given AZ could be easily handled by simply listing the subnets and finding where a given IP belongs.

Totally agree!

This service is published more as a concept to be built on top of, than a complete solution.

You wouldn't even need IAM rights to read RDS information, you need subnet information. As subnets are zonal, it does not if the service is RDS or Redis/ElastiCache. The IP returned from the hostname lookup, at the time your pod is scheduled, determines which AZ that Pod should (optimally) be deployed to.

Where this solution was created, was in a multi AWS account environment. Doing describe subnets API calls across multiple accounts is a hassle. It was "good enough" to have a static mapping of subnets, as they didn't change frequently.

Re: Automatic K8s pod placement to match external service zones

#17

Post by toredash »

stackskipton wrote: post #13
toredash wrote: post #2

Hi HN, I wanted to share something I've worked a bit to solve regarding Kubernetes: its scheduler has no awareness of the network topology for external services that workloads communicate with. If a pod talks to a database (e.g AWS RDS), K8s does not know it should schedule it in the same AZ as the database. If placed in the wrong AZ, it leads to unnecessary cross-AZ network traffic, adding latency (and costs $). I'v…

How do you handle RDS failovers? Mutating Webhook is only fired when Pods are created so if AZ zone does not fail, there is no pods to be created and affinity rules to be changed.

As it stands now, it doesn't. Unless you modify the Kyverno Policy to be of a background scanning.

I would create a similar policy where Kyverno at intervals would check the Deployment spec to see if the endpoint is changed, and alter the affinity rules. It would then be a traditional update of the Deployment spec to reflect the desire to run in another AZ, if that made sense?

Re: Automatic K8s pod placement to match external service zones

#18

Post by glennpratt »

ruuda wrote: post #8

> Have you considered alternative solutions? How about, don't use Kubernetes? The lack of control over where the workload runs is a problem caused by Kubernetes. If you deploy an application as e.g. systemd services, you can pick the optimal host for the workload, and it will not suddenly jump around.

Comparing systemd and Kubernetes for this scenario is like comparing an apple tree to a citrus grove.

You can specify just about anything, including exact nodes, for Kubernetes workloads.

This is just injecting some of that automatically.

I'm not knocking systemd, it's just not relevant.

Re: Automatic K8s pod placement to match external service zones

#19

Post by Spivak »

ruuda wrote: post #8

> Have you considered alternative solutions? How about, don't use Kubernetes? The lack of control over where the workload runs is a problem caused by Kubernetes. If you deploy an application as e.g. systemd services, you can pick the optimal host for the workload, and it will not suddenly jump around.

You need it to jump around because your RDS database might fail over to a different AZ.

Being able to move workloads around is kinda the point. The need exists irrespective of what you use to deploy your app.

Re: Automatic K8s pod placement to match external service zones

#20

Post by aduwah »

ruuda wrote: post #8

> Have you considered alternative solutions? How about, don't use Kubernetes? The lack of control over where the workload runs is a problem caused by Kubernetes. If you deploy an application as e.g. systemd services, you can pick the optimal host for the workload, and it will not suddenly jump around.

Mind you, that you are facing the same problem with any Autoscaling group that lives in multiple AZs. You don't need kubernetes for this
Post reply on HN