°whispers° cloud native...!
When AWS Autoscale Doesn’t
51–60 of 99 posts
Re: When AWS Autoscale Doesn’t
#52I got bit by this! Even worse is that one of the servers crumpled because we didn't scale up fast enough - so AWS killed it because of the health metric. Which then took out the remaining two because they were then far, far over capacity. I got the pager duty alert and found a total cluster and just manually set it to scale up way bigger. Now for all big events we manually bump minimum server counts for that period :\
Re: When AWS Autoscale Doesn’t
#53Vertical autoscale is on my wish list. Some way to automatically scale instance size for those things that don't scale well horizontally.
Jelastic claims to do this, and the marketing made it sound so cool. https://jelastic.com/
But when I tried it, it turns out the docker support requires very specific base images.
Not really docker then is it?!?
Re: When AWS Autoscale Doesn’t
#54Earlier quoted context omitted.
I think that Fargate is the “improvement” for ECS. I never understood the appeal of ECS in the first place, seemed (and still does) really half baked.
AWS employee here. Sorry to hear that you feel ECS is half baked. Feel free to reach out directly using the details in my profile info if you have any feedback you'd like me pass on to the team. To clear up the confusion on the relationship between Fargate and ECS, think of Fargate as the hosting layer: it runs your container for you on demand and bills you for the amount of CPU and GB your container reserved per sec…
From my perspective Fargate offers the functionality I would have expected from ECS in the first place. What ECS provides OOB requires too much janitoring and ultimately isn't terribly different in effort compared to running your own k8s or mesos infra on EC2 instances you provisioned yourself. You still basically needed an orchestration layer over ECS.
Which is why, I assume, Fargate is now listed as an integral feature of ECS on the product page.
Re: When AWS Autoscale Doesn’t
#55Earlier quoted context omitted.
AWS employee here. Sorry to hear that you feel ECS is half baked. Feel free to reach out directly using the details in my profile info if you have any feedback you'd like me pass on to the team. To clear up the confusion on the relationship between Fargate and ECS, think of Fargate as the hosting layer: it runs your container for you on demand and bills you for the amount of CPU and GB your container reserved per sec…
From my perspective Fargate offers the functionality I would have expected from ECS in the first place. What ECS provides OOB requires too much janitoring and ultimately isn't terribly different in effort compared to running your own k8s or mesos infra on EC2 instances you provisioned yourself. You still basically needed an orchestration layer over ECS. Which is why, I assume, Fargate is now listed as an integral fea…
Yeah to be clear the Fargate container hosting was always the vision for ECS, from the very first internal proposal to build this system. But its necessary to build something that keeps track of container state at scale first, and that is ECS. We built ECS so that it can keep track of container state both in Fargate and containers running on your own self managed EC2 hosts. This gives you the most flexibility if you have really specific needs for your container hosts that Fargate can't cover for you.
Re: When AWS Autoscale Doesn’t
#56Earlier quoted context omitted.
Joyent offers this as I recall, however it's only a scale up, I don't think they can scale down afterwards.
I can see the revenue issue with auto scale down, so I suppose that makes sense.
I recall it being technical actually, removing a proc from a running kernel isn’t trivial.
Re: When AWS Autoscale Doesn’t
#57Earlier quoted context omitted.
> Having two resources (such as DB and app) scale in concert can be exceedingly difficult. This means your resources are too tightly coupled. If they are so tightly coupled that they need to scale together then they are not two resources and you should look into restructuring them into two actual resources or bind them more closely to make a single resource.
In my example of DB and backend: How could they be decoupled?
As far as runtime, applications and DBs are already decoupled. You have N application instances mapped to M database instances. Applications can usually scale pretty much with load. Databases vary wildly in how they scale and it depends on the DB type.
Re: When AWS Autoscale Doesn’t
#58There are many limitations that you need to "read between the lines" with AWS auto scaling. For example, we have daemons reading messages from SQS, if you try to use auto scaling based on SQS metrics, you come to realize pretty quickly that CloudWatch is updated every 5 minutes. For most messages, this is simply too late. In a lot of cases, you are better off with updating CloudWatch yourself with your own interval u…
Out of curiosity, what’s the use case for running ECS on EC2 (instead of using Fargate) these days?
Apart from pricing and the potential to overcommit resources on EC2-ECS, there are a couple of other differences.
One is your options for doing forensics on Fargate. AWS manage the underlying host so you give up the option of Doing host level investigations. It’s not necessarily worse as you can fill this gap in other ways.
Logging is currently only via CloudWatch logs so if you want to get logs into something like Splunk you’ll have to run something that can pick up these logs. You’ll have that issue to solve if you want logs from some other AWS services like Lambda to go to the same place. The bigger issue for us is that you can’t add additional metadata to log events without building that into your application or getting tricky with log group names. On EC2 we’ve been using fluentd to add additional context to each log event like the instance it came from, the AZ, etc. Support for additional log drivers on Fargate is on the public roadmap[1][2] so there will hopefully be some more options soon.
[1] Fargate Log driver support v1 https://github.com/aws/containers-roadmap/issues/9 [2] Fargate log driver support v2 https://github.com/aws/containers-roadmap/issues/10
Re: When AWS Autoscale Doesn’t
#59I've also been deploying services on ECS for close to a year now and would like to address some inaccuracies the author seems to have made:
1) in 'Surprise 1' the author offers examples of CPU Utilization (or target) is between 80% and 95% without mentioning the reserved CPU/memory (aka size) of those tasks (under the assumption that he's using the Fargate launch type). The 'size' of a task also influences the average CPU target utilization. For instance, if a task requires the reserved CPU of 4 vCPUs, then a spike from 80% to 95% is handled differently than when a task reserves 1 or 2 vCPUs. The same goes for memory. In an example setup I'd use 1-2 vCPUs sized tasks with a service-wide target avg. CPU Utilization of 70% along and a StepScaling policy which adds 10% more tasks if the service avg. CPUU falls between 70-80, 20% if between 80-90 and 25% if above 90. My strategy has been being smaller-sized tasks, lower service avg CPU utilization (compared to 80%-90%) and shorter evaluation periods/datapoints for the scale-out CW alarms (minimum being 60 seconds IIRC). The short evaluation periods/low number of datapoints of the CW alarm allowed me to handle spikes reasonably fast.
2) in 'Surprise 3' the author claims that the Terraform's aws_appautoscaling_policy 'is rather light on documentation'. Since I am a user of Terraform for several years, I find it inaccurate mostly because of the several examples available in the documentation https://www.terraform.io/docs/providers/aws/r/appautoscaling... as well as many more when doing a Github exact search for "aws_appautoscaling_policy" language:HCL will reveal many, many more examples from open-source repos (some with permissive licenses too). I'd created a custom ecs-service TF module which creates for each service (optionally) an ALB along with listeners and the attached ACM-issued TLS certs and TGs, the scale-in/out CW alerts with configurable thresholds/policies, SGs, Route53, etc. allowing one to quickly configure and launch an ECS service fast and reliably.
Regarding the scale-in, I typically also have that at intervals between 5-15 minutes to avoid an erratic scale-in/scale-out 'zig-zag' happening even at the cost of briefly over provisioning.
Re: When AWS Autoscale Doesn’t
#60The way I've been happiest using EC2 Auto Scaling was to have a single cron-job continuously calculating how many instances I should be running, and it sets the desired capacity manually with the Auto Scaling API[1]. This may seem to defeat the purpose of Auto Scaling, but it's actually much more convenient than spinning up/down EC2 instances with the EC2 API. You get to precisely control how to scale, and won't be a…
So we had this cause a spectacular outage a few years ago. We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down. So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually laun…
I don't understand--how were you launching instances if the API(s) was/were down? Your system was unable to determine that there were instances running, but it was able to send RunInstances requests to EC2?