Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Automatic K8s pod placement to match external service zones

github.com

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Automatic K8s pod placement to match external service zones

#41

Post by toredash »

solatic wrote: post #28

I don't really understand why you think this tool is needed and what exact problem/risk it's trying to solve. Most people should start with a single-zone setup and just accept that there's a risk associated with zone failure. If you have a single-zone setup, you have a node group in that one zone, you have the managed database in the same zone, and you're done. Zone-wide failure is extremely rare in practice and you…

> Most people should start with a single-zone setup and just accept that there's a risk associated with zone failure. If you have a single-zone setup, you have a node group in that one zone, you have the managed database in the same zone, and you're done.

I don't disagree, but there is one issue with this approach and that is that RDS is a multi AZ service by itself. That means that when a maintenance event occur on your insaance, AWS will start a new instance in a new zone, and fail over to that one.

You could of course manually failover RDS afterwards to your primary zone. Not sure if that is better than manually scaling up a node pool if a zone fails.

> So you are presuming that, when RDS automatically fails over to zone b to account for zone a failure, that you will certainly be able to scale up a full scale production environment in zone b as well, in spite of nearly every other AWS customer attempting more or less the same strategy;

Thats up to the user to decide via the Kyverno policy. We used the preferredDuringSchedulingIgnoredDuringExecution affinity setting to instruct the scheduler to attempt to schedule the pods in the optimal zone.

I believe the only way to be 100% sure that you have compute capacity available in your AWS account is the use EC2 On-Demand Capacity Reservations (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capa...). If your current zone is at full capacity, and for some reason the nodes your VMs are running on dies, that capacity is lost, and you wont get it back either.

Re: Automatic K8s pod placement to match external service zones

#42

Post by solatic »

toredash wrote: post #41
solatic wrote: post #28

I don't really understand why you think this tool is needed and what exact problem/risk it's trying to solve. Most people should start with a single-zone setup and just accept that there's a risk associated with zone failure. If you have a single-zone setup, you have a node group in that one zone, you have the managed database in the same zone, and you're done. Zone-wide failure is extremely rare in practice and you…

> Most people should start with a single-zone setup and just accept that there's a risk associated with zone failure. If you have a single-zone setup, you have a node group in that one zone, you have the managed database in the same zone, and you're done. I don't disagree, but there is one issue with this approach and that is that RDS is a multi AZ service by itself. That means that when a maintenance event occur on…

> That means that when a maintenance event occur on your insaance, AWS will start a new instance in a new zone, and fail over to that one.

Not true for single-AZ deployments. There is downtime during the maintenance event, but this is also true in multi-AZ deployments when the instance in the second AZ is promoted; a multi-AZ maintenance window has slightly less downtime, but not much; downtime is downtime, but generally not enough to affect a 99.9% SLA anyway.

> EC2 On-Demand Capacity Reservations

Also quite expensive to maintain just for outage recovery events.

The point I'm trying to make is that formal risk analysis forces you to think about actual sources of risk, and SRE/FinOps principles force you think about how much budget you are willing to spend to address those risks. And I don't understand how a tool like this fits into formal risk analysis and where it presents an optimum solution for those risks.

Re: Automatic K8s pod placement to match external service zones

#43

Post by toredash »

solatic wrote: post #42
toredash wrote: post #41

Earlier quoted context omitted.

> Most people should start with a single-zone setup and just accept that there's a risk associated with zone failure. If you have a single-zone setup, you have a node group in that one zone, you have the managed database in the same zone, and you're done. I don't disagree, but there is one issue with this approach and that is that RDS is a multi AZ service by itself. That means that when a maintenance event occur on…

> That means that when a maintenance event occur on your insaance, AWS will start a new instance in a new zone, and fail over to that one. Not true for single-AZ deployments. There is downtime during the maintenance event, but this is also true in multi-AZ deployments when the instance in the second AZ is promoted; a multi-AZ maintenance window has slightly less downtime, but not much; downtime is downtime, but gener…

> And I don't understand how a tool like this fits into formal risk analysis and where it presents an optimum solution for those risks.

Seems it does not fit your risk analysis?

Re: Automatic K8s pod placement to match external service zones

#44

Post by mathverse »

toredash wrote: post #11
dserodio wrote: post #7

Earlier quoted context omitted.

It's a best practice to have a Deployment run multiple Pods in separate AZs to increase availability

Yes I get that. But are we talking HA for this lookup service that I've made? If yes, that's a simple update of the manifest to have 3 replicas with ab affinity setting to spread that out over different AZ. Kyverno would use the internal Service object this service provide to have a HA endpoint to send queries to. If we are not talking about this AZP service, I don't understand what we are talkin about.

If you have the setup on 3 AZs how would you route traffic only to the AZ where your RDS resides?

Re: Automatic K8s pod placement to match external service zones

#45

Post by toredash »

mathverse wrote: post #44
toredash wrote: post #11

Earlier quoted context omitted.

Yes I get that. But are we talking HA for this lookup service that I've made? If yes, that's a simple update of the manifest to have 3 replicas with ab affinity setting to spread that out over different AZ. Kyverno would use the internal Service object this service provide to have a HA endpoint to send queries to. If we are not talking about this AZP service, I don't understand what we are talkin about.

If you have the setup on 3 AZs how would you route traffic only to the AZ where your RDS resides?

> If you have the setup on 3 AZs how would you route traffic only to the AZ where your RDS resides?

So specifically for RDS, AWS will provide two endpoint for the client application: A writer and a reader endpoint. Similar to this: mydbcluster.cluster-c7tj4example.us-east-1.rds.amazonaws.com : Writer endpoint mydbcluster.cluster-ro-c7tj4example.us-east-1.rds.amazonaws.com : Reader endpoint (notice -ro part).

The writer endpoint will always resolve to the active master, which is what the client application is configured to use, and thats the hostname my lookup service will use as input to determine the current location of the Writer instance.

My solution works only for hostnames that returns a single IP address, so it won't work for the Reader endpoints. As I wrote in the repository, a requirement for this is that "The FQDN needs to return a single A record for the external resource".

Re: Automatic K8s pod placement to match external service zones

#46

Post by westurner »

toredash wrote: post #40
westurner wrote: post #31

Couldn't something like this make CI builds faster by running builds near already-cached container images?

Are you thinking about already-cached container images on the host level ? Not sure how AZP fits in here? Since you mentioned it, what I've done before when it comes to improving CI builds, is to use karpenter + local SSD mounts with very large instance types in an idle timeout of ~1h. This allowed us to have very performant build machines at a low cost. The first build of the day took a while to get going, but for t…

Are the container image repositories and the container images also "external resources" that could make CI build pod placement more efficient?

Thanks; that sounds faster than most self-hosted CI services.

Re: Automatic K8s pod placement to match external service zones

#47

Post by toredash »

westurner wrote: post #46
toredash wrote: post #40

Earlier quoted context omitted.

Are you thinking about already-cached container images on the host level ? Not sure how AZP fits in here? Since you mentioned it, what I've done before when it comes to improving CI builds, is to use karpenter + local SSD mounts with very large instance types in an idle timeout of ~1h. This allowed us to have very performant build machines at a low cost. The first build of the day took a while to get going, but for t…

Are the container image repositories and the container images also "external resources" that could make CI build pod placement more efficient? Thanks; that sounds faster than most self-hosted CI services.

If the image repositories were AZ bound resources, that would make the CI build process more efficient.

Or, if the resources that CI build is utilizing within the image (after the image is pulled and started) is AZ bound, then yes the build process would be improved since the CI build would fetch AZ local resources, rather than crossing the AZ boundary

Post reply on HN