Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When AWS Autoscale Doesn’t

segment.com

Post reply on HN
Print view View original

71–80 of 99 posts

Re: When AWS Autoscale Doesn’t

#71

Post by si-ruslan »

etaioinshrdlu wrote: post #53
tyingq wrote: post #5

Vertical autoscale is on my wish list. Some way to automatically scale instance size for those things that don't scale well horizontally.

Jelastic claims to do this, and the marketing made it sound so cool. https://jelastic.com/ But when I tried it, it turns out the docker support requires very specific base images. Not really docker then is it?!?

Hi, Jelastic founder is here. Thank you for mentioning our product. Vertical scaling is not a marketing :), it's reality. Only Jelastic offers automatic vertical scaling with pay-as-you-use billing model. There is a high level overview of the pay-as-you-go vs pay-as-you-use https://www.forbes.com/sites/forbestechcouncil/2018/03/28/de.... Except this we help related technologies to become more elastic, for example Java https://jelastic.com/blog/elastic-jvm-vertical-scaling/

Re: When AWS Autoscale Doesn’t

#72

Post by viraptor »

wahnfrieden wrote: post #67
eridius wrote: post #66

> For example, the maximum value for CPU utilization that you can have regardless of load is 100%. I'm surprised it can't use load average to estimate the true resource demand.

You can, but it’s not visible to the hypervisor (it’s an OS concept) so you have to publish that metric from an agent on the machine. Then you can use it for autoscaling.

But even then, "load" doesn't work well for all workloads. http://www.brendangregg.com/blog/2017-08-08/linux-load-avera...

The number of switching tasks may be very high for a number of reasons, including a very large number of threads which do very small chunk of work each and yield.

Re: When AWS Autoscale Doesn’t

#73

Post by si-ruslan »

etaioinshrdlu wrote: post #53
tyingq wrote: post #5

Vertical autoscale is on my wish list. Some way to automatically scale instance size for those things that don't scale well horizontally.

Jelastic claims to do this, and the marketing made it sound so cool. https://jelastic.com/ But when I tried it, it turns out the docker support requires very specific base images. Not really docker then is it?!?

Regarding the docker support, there are two flavors inside Jelastic:

1) Native Docker Engine - you can create a dedicated container engine for your project in the same way as you do on any IaaS today, for example

How to run Docker Swarm https://jelastic.com/blog/docker-swarm-auto-clustering-and-s... How to run Docker Engine https://jelastic.com/blog/docker-engine-automatic-install-sw...

The advantage here is the exactly vertical scaling feature. In Jelastic unused resources will not be considered as paid while at any other cloud provider you will have to pay for the resource limits.

2) Enhanced System Containers based on Dockerfile - there is no need to provision a dedicated docker engine or swarm. This solution provides even better density, elasticity, multi-tenancy and security, more advanced integration with UI and PaaS features set compared to #1. It supports multiple processes inside a single container, you can get an SSH access and use all standard tools for app deployment, write to local filesystem, use multicast and so on. It supports traditional or legacy apps while images can be prepared in the same familiar Dockefile format. Unfortunately it's not fully compatible with Native Docker Engine due to specifics limitations/requirements of docker technology itself.

Thank you for pointing out this issue. In the upcoming release we will clarify the difference between two and provide more tips which one is better to use in various cases.

Re: When AWS Autoscale Doesn’t

#74

Post by samstave »

aequitas wrote: post #61
samstave wrote: post #33

Earlier quoted context omitted.

So we had this cause a spectacular outage a few years ago. We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down. So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually laun…

AWS has limits on the amount of resources you can have in a VPC. You can request to increase these via a process out of bounds of the API. This mechanism is there exactly for these kind of things (and malicious API calls should you get hacked). Maybe someone at your company was thinking to big? Normally these are around 10/50 for each EC2 type.

you are aware that if you have a close enough relationship with AWS you can request and set your own limits?

Limits are malleable based on your use case. Speak with your rep.

You might even not know how limits came to be... I am.

---

I have had a time when git suffered a flaw, and a junior dev also suffered a flaw in checking in secrets.... thousands of instances across the globe were launched... for bitcoin mining... $700,000 in a few hours...

We all learned a bit that day.

Re: When AWS Autoscale Doesn’t

#75

Post by auslander »

> .. the ECS dashboard does not yet support .. Terraform ..

They haven't yet matured enough, it seems. Cloudformation is the right way to code your infra, not web Console or anything else.

Good sign, though, is they use ECS, not Kubernetes :)

Re: When AWS Autoscale Doesn’t

#76

Post by anbotero »

Quick question for the AWS employee solving inquiries: I used ECS in 2017, and back then there was this weird issue where sometimes tasks would switch to new versions in like, a minute (if that), but sometimes, like 2/10, it would take like 10-12 minutes just for it to start killing old Task containers. Back then there wasn't any timeout option or anything to force the killing. Do you know if now there is? The project was killed for different reasons, but I really liked everything else on ECS. Thanks!

EDIT: I meant killing containers, not the Tasks themselves. Sorry.

Re: When AWS Autoscale Doesn’t

#77

Post by meekins »

thecopy wrote: post #8
doctorpangloss wrote: post #6

An incredible amount of software and infrastructure is written precisely for analytics data gathering workloads. I'm pretty confident AWS's product for this use case would be Lambda and the new on-demand DynamoDB. Is there actually a use case in analytics that requires a server that accepts connections from multiple clients, and then has to have Because if there isn't, if your goal is to scale, why have containers at…

Batching incoming requests, for one. Kinesis only allows 5 write requests per second per shard, for example. As well, Lambda have limits regarding concurrent executions and are very slow (10s) if needing VPC connectivity (in this case the default concurrent lambda limit is 350 due to ENIs)

Hmm... I don't see anything in the docs implying that - Kinesis API docs say it's possible to ingest 1000 records or 1MB per shard per second. There's a 5/s limit on reads however but those deal with batches of records anyway.

We have one service running that consumes data to a Kinesis stream published as an API GW endpoint. Preprocessing is done in Lambda in batches of 100 records and the processed records get pushed to Firehose streams for batched loads to a Redshift cluster for analytics. So far we've been very happy with the solution - very little custom code, almost no ops required and it performs and scales well.

Re: When AWS Autoscale Doesn’t

#78

Post by AmericanChopper »

samstave wrote: post #33
DVassallo wrote: post #16

The way I've been happiest using EC2 Auto Scaling was to have a single cron-job continuously calculating how many instances I should be running, and it sets the desired capacity manually with the Auto Scaling API[1]. This may seem to defeat the purpose of Auto Scaling, but it's actually much more convenient than spinning up/down EC2 instances with the EC2 API. You get to precisely control how to scale, and won't be a…

So we had this cause a spectacular outage a few years ago. We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down. So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually laun…

Another thing to keep in mind is that AWS local capacity can run incredibly close to the wire at times. You might be surprised if you knew how much capacity for your instance type was actually available under the hood. I’ve personally seen insufficient capacity errors.

Re: When AWS Autoscale Doesn’t

#79

Post by zepolen »

icelancer wrote: post #64
DVassallo wrote: post #38

Earlier quoted context omitted.

That's not the right way to do it. You shouldn't monitor how many instances you're running. You just need to determine how many instances you should be running based on your scaling driver (cpu, # of users, database connections, etc). Then you call the Auto Scaling SetDesiredCapacity API with the number, and it is idempotent[1]. If the AWS API is down, your fleet size just won't change. [1] https://docs.aws.amazon.co…

> That's not the right way to do it. The poster is aware of this, which is why they talked specifically what they did wrong.

While the poster was aware of it, he did not provide a solution whereas DVassallo provided a valuable step by step on how to do it properly. Which may help others in the future.

Think long and hard why you felt it necessary to make your comment and what value it actually provided.

Re: When AWS Autoscale Doesn’t

#80

Post by samstave »

zepolen wrote: post #79
icelancer wrote: post #64

Earlier quoted context omitted.

> That's not the right way to do it. The poster is aware of this, which is why they talked specifically what they did wrong.

While the poster was aware of it, he did not provide a solution whereas DVassallo provided a valuable step by step on how to do it properly. Which may help others in the future. Think long and hard why you felt it necessary to make your comment and what value it actually provided.

If this is still a pitfall for users of AWS ~5 years in.. then its not a fault of my communication...

You know what I think its a fault of:

Lack of a canonical DevOps "university" stemmed from SV startups.

DevOps at this point should not just be a degree -- it should be a freaking field of study and classes offered by YC.... Look at their pedigree of companies at scale. We should all make an effort to give back in this regard...

Post reply on HN