Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When AWS Autoscale Doesn’t

segment.com

Post reply on HN
Print view View original

31–40 of 99 posts

Re: When AWS Autoscale Doesn’t

#31

Post by etaioinshrdlu »

eunoia wrote: post #11
avitzurel wrote: post #2

There are many limitations that you need to "read between the lines" with AWS auto scaling. For example, we have daemons reading messages from SQS, if you try to use auto scaling based on SQS metrics, you come to realize pretty quickly that CloudWatch is updated every 5 minutes. For most messages, this is simply too late. In a lot of cases, you are better off with updating CloudWatch yourself with your own interval u…

Out of curiosity, what’s the use case for running ECS on EC2 (instead of using Fargate) these days?

Perhaps cost?

Re: When AWS Autoscale Doesn’t

#32

Post by jusssi »

eunoia wrote: post #11
avitzurel wrote: post #2

There are many limitations that you need to "read between the lines" with AWS auto scaling. For example, we have daemons reading messages from SQS, if you try to use auto scaling based on SQS metrics, you come to realize pretty quickly that CloudWatch is updated every 5 minutes. For most messages, this is simply too late. In a lot of cases, you are better off with updating CloudWatch yourself with your own interval u…

Out of curiosity, what’s the use case for running ECS on EC2 (instead of using Fargate) these days?

At least one is, that you get to use the leftover CPU and memory for your other containers when you use an EC2 instance. With some workloads this lets you to overcommit those resources if you know all your containers won't max out simultaneously.

Edit: another one is that you can run ECS on spot fleet and save some money.

Re: When AWS Autoscale Doesn’t

#33

Post by samstave »

DVassallo wrote: post #16

The way I've been happiest using EC2 Auto Scaling was to have a single cron-job continuously calculating how many instances I should be running, and it sets the desired capacity manually with the Auto Scaling API[1]. This may seem to defeat the purpose of Auto Scaling, but it's actually much more convenient than spinning up/down EC2 instances with the EC2 API. You get to precisely control how to scale, and won't be a…

So we had this cause a spectacular outage a few years ago.

We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down.

So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually launching instances.

The increased scale of instances pummeled the control plane with thousands of instances all trying to come online and pull down their needed data to get operational -- which them killed our DBs, pipeline etc...

We had to reboot our entire production environment at peak service time...

Re: When AWS Autoscale Doesn’t

#34

Post by nicoburns »

samstave wrote: post #33
DVassallo wrote: post #16

The way I've been happiest using EC2 Auto Scaling was to have a single cron-job continuously calculating how many instances I should be running, and it sets the desired capacity manually with the Auto Scaling API[1]. This may seem to defeat the purpose of Auto Scaling, but it's actually much more convenient than spinning up/down EC2 instances with the EC2 API. You get to precisely control how to scale, and won't be a…

So we had this cause a spectacular outage a few years ago. We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down. So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually laun…

What was your resolution to this issue? Did you fix your service to account for the API being down, or did you switch to an entirely different approach?

Re: When AWS Autoscale Doesn’t

#35

Post by tyingq »

xorgar831 wrote: post #30
barbecue_sauce wrote: post #18

Earlier quoted context omitted.

I didn't even realize this wasn't part of the offering yet.

Joyent offers this as I recall, however it's only a scale up, I don't think they can scale down afterwards.

I can see the revenue issue with auto scale down, so I suppose that makes sense.

Re: When AWS Autoscale Doesn’t

#36

Post by samstave »

nicoburns wrote: post #34
samstave wrote: post #33

Earlier quoted context omitted.

So we had this cause a spectacular outage a few years ago. We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down. So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually laun…

What was your resolution to this issue? Did you fix your service to account for the API being down, or did you switch to an entirely different approach?

I can't recall the exact implementation detail, but We then logged the number of running instances in a file, and read the last qty of instances and the delta from when launched - and made the system not get over aggressive if it couldnt read the current set.

We also added smart loading across AZs due to spot instances getting whacked when our fleet was outbid and AWS took them back.

As well as other monitoring methods to be sure we werent caught with a smart system doing dumb things.

Re: When AWS Autoscale Doesn’t

#37

Post by NathanKP »

acd wrote: post #19

AWS needs to glue EC2 and ECS scheduling together. Today the schedulers are separate. So basically the feet does not know what the arms are doing. That leaves fixing this scaling up to the client meaning duplicate code effort solving the same thing for each AWS customer.

AWS employee here.

This is a feature that is currently on our public roadmap for container services, in the "Researching" category: https://github.com/aws/containers-roadmap/issues/76

Feel free to drop a thumbs up on the roadmap item to show your support and boost its priority on the roadmap, or leave a comment to let us know more about your needs.

Re: When AWS Autoscale Doesn’t

#38

Post by DVassallo »

samstave wrote: post #33
DVassallo wrote: post #16

The way I've been happiest using EC2 Auto Scaling was to have a single cron-job continuously calculating how many instances I should be running, and it sets the desired capacity manually with the Auto Scaling API[1]. This may seem to defeat the purpose of Auto Scaling, but it's actually much more convenient than spinning up/down EC2 instances with the EC2 API. You get to precisely control how to scale, and won't be a…

So we had this cause a spectacular outage a few years ago. We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down. So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually laun…

That's not the right way to do it. You shouldn't monitor how many instances you're running. You just need to determine how many instances you should be running based on your scaling driver (cpu, # of users, database connections, etc). Then you call the Auto Scaling SetDesiredCapacity API with the number, and it is idempotent[1]. If the AWS API is down, your fleet size just won't change.

[1] https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API...

Re: When AWS Autoscale Doesn’t

#39

Post by NathanKP »

pram wrote: post #9
Legogris wrote: post #7

Having used ECS quite a bit, I do not recommend anyone building a new stack based on it. Kubernetes solves everything ECS solves, but usually better and without sveral of the issues mentioned here. Last time I checked, AWS was still lagging behind Azure and GCP on Kubernetes, but I have a strong feeling they're prioritizing improving EKS over ECS. If you're already invested in ECS it's a different story, of course.

I think that Fargate is the “improvement” for ECS. I never understood the appeal of ECS in the first place, seemed (and still does) really half baked.

AWS employee here. Sorry to hear that you feel ECS is half baked. Feel free to reach out directly using the details in my profile info if you have any feedback you'd like me pass on to the team.

To clear up the confusion on the relationship between Fargate and ECS, think of Fargate as the hosting layer: it runs your container for you on demand and bills you for the amount of CPU and GB your container reserved per second. On the other hand ECS is the management layer. It provides the API that you use to orchestrate launching X containers, spreading them across availability zones, and hooking them up to other resources automatically (like load balancers, service discovery, etc).

Currently you can use ECS without using Fargate, by providing your own pool of EC2 instances to host the containers on. However, you can not use Fargate without ECS, as the hosting layer doesn't know how to run your full application stack without being instructed to by the ECS management layer.

Re: When AWS Autoscale Doesn’t

#40

Post by matwood »

eunoia wrote: post #11
avitzurel wrote: post #2

There are many limitations that you need to "read between the lines" with AWS auto scaling. For example, we have daemons reading messages from SQS, if you try to use auto scaling based on SQS metrics, you come to realize pretty quickly that CloudWatch is updated every 5 minutes. For most messages, this is simply too late. In a lot of cases, you are better off with updating CloudWatch yourself with your own interval u…

Out of curiosity, what’s the use case for running ECS on EC2 (instead of using Fargate) these days?

A large part of our ECS capacity is running on spot instances which are much cheaper.
Post reply on HN