Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When AWS Autoscale Doesn’t

segment.com

Post reply on HN
Print view View original

61–70 of 99 posts

Re: When AWS Autoscale Doesn’t

#61

Post by aequitas »

samstave wrote: post #33
DVassallo wrote: post #16

The way I've been happiest using EC2 Auto Scaling was to have a single cron-job continuously calculating how many instances I should be running, and it sets the desired capacity manually with the Auto Scaling API[1]. This may seem to defeat the purpose of Auto Scaling, but it's actually much more convenient than spinning up/down EC2 instances with the EC2 API. You get to precisely control how to scale, and won't be a…

So we had this cause a spectacular outage a few years ago. We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down. So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually laun…

AWS has limits on the amount of resources you can have in a VPC. You can request to increase these via a process out of bounds of the API. This mechanism is there exactly for these kind of things (and malicious API calls should you get hacked). Maybe someone at your company was thinking to big? Normally these are around 10/50 for each EC2 type.

Re: When AWS Autoscale Doesn’t

#62

Post by samstave »

DVassallo wrote: post #38
samstave wrote: post #33

Earlier quoted context omitted.

So we had this cause a spectacular outage a few years ago. We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down. So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually laun…

That's not the right way to do it. You shouldn't monitor how many instances you're running. You just need to determine how many instances you should be running based on your scaling driver (cpu, # of users, database connections, etc). Then you call the Auto Scaling SetDesiredCapacity API with the number, and it is idempotent[1]. If the AWS API is down, your fleet size just won't change. [1] https://docs.aws.amazon.co…

Yeah, like I said - this was a few years ago, and the system wasnt designed to be able to scale using ASGs at the time. (Fleet didnt yet exist, amd a bunch of other reasons) - but scaling was based on users and load complexity for the data we were handling -- this wasnt a web service.

Re: When AWS Autoscale Doesn’t

#63

Post by tudelo »

robrtsql wrote: post #60
samstave wrote: post #33

Earlier quoted context omitted.

So we had this cause a spectacular outage a few years ago. We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down. So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually laun…

I don't understand--how were you launching instances if the API(s) was/were down? Your system was unable to determine that there were instances running, but it was able to send RunInstances requests to EC2?

Queue them up?

Re: When AWS Autoscale Doesn’t

#64

Post by icelancer »

DVassallo wrote: post #38
samstave wrote: post #33

Earlier quoted context omitted.

So we had this cause a spectacular outage a few years ago. We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down. So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually laun…

That's not the right way to do it. You shouldn't monitor how many instances you're running. You just need to determine how many instances you should be running based on your scaling driver (cpu, # of users, database connections, etc). Then you call the Auto Scaling SetDesiredCapacity API with the number, and it is idempotent[1]. If the AWS API is down, your fleet size just won't change. [1] https://docs.aws.amazon.co…

> That's not the right way to do it.

The poster is aware of this, which is why they talked specifically what they did wrong.

Re: When AWS Autoscale Doesn’t

#65

Post by samstave »

robrtsql wrote: post #60
samstave wrote: post #33

Earlier quoted context omitted.

So we had this cause a spectacular outage a few years ago. We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down. So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually laun…

I don't understand--how were you launching instances if the API(s) was/were down? Your system was unable to determine that there were instances running, but it was able to send RunInstances requests to EC2?

Correct.

Couldn't query, but could initiate.

Re: When AWS Autoscale Doesn’t

#67

Post by wahnfrieden »

eridius wrote: post #66

> For example, the maximum value for CPU utilization that you can have regardless of load is 100%. I'm surprised it can't use load average to estimate the true resource demand.

You can, but it’s not visible to the hypervisor (it’s an OS concept) so you have to publish that metric from an agent on the machine. Then you can use it for autoscaling.

Re: When AWS Autoscale Doesn’t

#68

Post by [deleted] »

DVassallo wrote: post #16

The way I've been happiest using EC2 Auto Scaling was to have a single cron-job continuously calculating how many instances I should be running, and it sets the desired capacity manually with the Auto Scaling API[1]. This may seem to defeat the purpose of Auto Scaling, but it's actually much more convenient than spinning up/down EC2 instances with the EC2 API. You get to precisely control how to scale, and won't be a…

[deleted]

Re: When AWS Autoscale Doesn’t

#70

Post by jugg1es »

Legogris wrote: post #7

Having used ECS quite a bit, I do not recommend anyone building a new stack based on it. Kubernetes solves everything ECS solves, but usually better and without sveral of the issues mentioned here. Last time I checked, AWS was still lagging behind Azure and GCP on Kubernetes, but I have a strong feeling they're prioritizing improving EKS over ECS. If you're already invested in ECS it's a different story, of course.

Agreed. ECS has several limitations that you don't really discover until you are fully into the weeds. Don't use it if you are just starting with AWS unless your use case is a straightforward website stack. Do not use it for complex microservice architectures.
Post reply on HN