Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Under the Hood of Amazon EC2 Container Service

allthingsdistributed.com

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Under the Hood of Amazon EC2 Container Service

#21

Post by djb_hackernews »

samstave wrote: post #20
Pyxl101 wrote: post #19

Earlier quoted context omitted.

> Right now - you have to manually [...] scale the ASG. You have to manually scale the AutoScaling Group, you say? :-) More seriously, what's the barrier to configuring the ASG to grow and shrink automatically?

Heh, yeah basically you can create an asg, but ecs has no trigger to scale it based on the number of task containers you attempt to launch on the cluster.... There is a workaround, apparently, which is to create a custom metric but aws says this has not been tested to their knowledge... So any asg will be "static" implicitly...

That is definitely inaccurate. We have asgs that scale based on cpu load and custom cloudwatch metrics.

With ecs I can imagine a metric that keeps track of the number of tasks and hosts, or ports used or something. I think asg is the perfect tool to use to auto scale ecs clusters.

Re: Under the Hood of Amazon EC2 Container Service

#22

Post by werner »

samstave wrote: post #15

The biggest issue that I had with ECS is that you need to initially create EC2 instances to put into your ECS cluster, using the AMI such that they have the ECS agent on them... BUT you have to prescale that cluster manually thereafter. In the task definition - it would be MUCH better if you could select the EC2 instance type you want, collect them in an ASG and have the task slicing scale the ASG accordingly. Right…

Nope, we have built a new scheduler for you that will allow placement over multiple AZ's, replace failed containers, allow them to connect to ELB's, etc.

Re: Under the Hood of Amazon EC2 Container Service

#23

Post by werner »

larsmak wrote: post #4
justinsb wrote: post #2

I think it is interesting that AWS seems to be moving to consistent data-stores. Previously they were championing eventual consistency everywhere, even when it made for painful products (SimpleDB) or painful APIs (retry loops when using EC2 APIs).

I don't know, S3 and DynamoDB are both eventual consistent. And keeping in mind the CAP-theorem it makes sense. And I for one love SimpleDB - it's just that, simple. And great for prototyping (really cheap) and small production-loads. Often you just need a place to stick your data, scalability can be achieved to adding a caching layer.

You can choose eventual or fully consistent in DynamoDB. Given that full consistency comes at a higher cost (read from a quorum of replicas) we expose that cost to you.

BTW nobody wants eventual consistency, it is a fact of live among many trade-offs. I would rather not expose it but it comes with other advantages ...

Re: Under the Hood of Amazon EC2 Container Service

#24

Post by noelwelsh »

justinsb wrote: post #2

I think it is interesting that AWS seems to be moving to consistent data-stores. Previously they were championing eventual consistency everywhere, even when it made for painful products (SimpleDB) or painful APIs (retry loops when using EC2 APIs).

I believe raw eventual consistency has failed as a programming API. I believe CRDTs in their many incarnations provide a great alternative but i) CRDTs are quite new and ii) they require a more complex API.

Re: Under the Hood of Amazon EC2 Container Service

#25

Post by 2461001642 »

werner wrote: post #22
samstave wrote: post #15

The biggest issue that I had with ECS is that you need to initially create EC2 instances to put into your ECS cluster, using the AMI such that they have the ECS agent on them... BUT you have to prescale that cluster manually thereafter. In the task definition - it would be MUCH better if you could select the EC2 instance type you want, collect them in an ASG and have the task slicing scale the ASG accordingly. Right…

Nope, we have built a new scheduler for you that will allow placement over multiple AZ's, replace failed containers, allow them to connect to ELB's, etc.

Thanks for responding! Can you say how you deal with the significant latencies over multiple AZs?

Also, would you mind answering my other question about whether or not I can run Marathon or Chronos against ECS since it runs Mesos under the hood?

Re: Under the Hood of Amazon EC2 Container Service

#26

Post by werner »

2461001642 wrote: post #25
werner wrote: post #22

Earlier quoted context omitted.

Nope, we have built a new scheduler for you that will allow placement over multiple AZ's, replace failed containers, allow them to connect to ELB's, etc.

Thanks for responding! Can you say how you deal with the significant latencies over multiple AZs? Also, would you mind answering my other question about whether or not I can run Marathon or Chronos against ECS since it runs Mesos under the hood?

No mesos under the hood. You can bring mesos as your own scheduler.

Re: Under the Hood of Amazon EC2 Container Service

#27

Post by werner »

2461001642 wrote: post #25
werner wrote: post #22

Earlier quoted context omitted.

Nope, we have built a new scheduler for you that will allow placement over multiple AZ's, replace failed containers, allow them to connect to ELB's, etc.

Thanks for responding! Can you say how you deal with the significant latencies over multiple AZs? Also, would you mind answering my other question about whether or not I can run Marathon or Chronos against ECS since it runs Mesos under the hood?

Can you elaborate on the intra- inter-* latencies your are seeing? for AZ independent services this should single digits.

Re: Under the Hood of Amazon EC2 Container Service

#28

Post by Rapzid »

A glaring gap currently is security. Per-container IAM roles would go a long way IMHO, but that still leaves "other" secret management which is a PITA. Other options such as kubernetes lack the AWS/ELB integration; all seem to be lacking a good security management model.

Re: Under the Hood of Amazon EC2 Container Service

#29

Post by 2461001642 »

werner wrote: post #26
2461001642 wrote: post #25

Earlier quoted context omitted.

Thanks for responding! Can you say how you deal with the significant latencies over multiple AZs? Also, would you mind answering my other question about whether or not I can run Marathon or Chronos against ECS since it runs Mesos under the hood?

No mesos under the hood. You can bring mesos as your own scheduler.

Thanks again - this is really helpful. I had talked to someone who had left Amazon but knew the internal workings who said ECS was Mesos just privately branded like Chef -> OpsWorks, but I guess I must have misunderstood.

Thanks for clarifying!

Re: Under the Hood of Amazon EC2 Container Service

#30

Post by 2461001642 »

werner wrote: post #27
2461001642 wrote: post #25

Earlier quoted context omitted.

Thanks for responding! Can you say how you deal with the significant latencies over multiple AZs? Also, would you mind answering my other question about whether or not I can run Marathon or Chronos against ECS since it runs Mesos under the hood?

Can you elaborate on the intra- inter-* latencies your are seeing? for AZ independent services this should single digits.

When I'm balancing a single deployment across multiple AZs (e.g. US-East -> US-West), the latencies between the containers seem far higher than just the 200-300ms predicted by speed of light. Am I doing something wrong?
Post reply on HN