Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Autoscaling on AWS

qzaidi.github.io

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Autoscaling on AWS

#21

Post by rschmitty »

mathrawka wrote: post #12

I have found that using a ubuntu base AMI and salt ( http://saltstack.com/community.html ) makes things much easier to manage. His suggestion of having node.js and nginx baked into the AMI is a big pain when you need to upgrade node.js or nginx (hello security update!). Using something like salt (or puppet or chef) would make this much easier. But in general, I agree with the article.

Yes it is a pain to bake your AMI that much, but the upside is you autoscale faster. When you can scale faster you can set your thresholds higher because you know a new instance will be ready to serve traffic through ELB faster.

Granted node and nginx don't take long to run an update on. But the more you keep piling on updates the slower your true "time to ready" increases. And that can take a while from ec2 launch to ELB registration.

Why not have best of both and use puppet/chef to create your AMI anytime they require a security update?

Re: Autoscaling on AWS

#22

Post by randallsquared »

contingencies wrote: post #17
morganK wrote: post #16

Earlier quoted context omitted.

Not that simple. Having only one cloud provider allow you to make the most of it. Once you have two providers, you are forced to only use the features offered by each of them and use an abstract layer (API agnostic dev). Plus, AWS offers a wide range of strategies to ensure the availability of its infrastructure (Availability Zone, Regions, CDN etc.)

Once you have two providers, you are forced to only use the features offered by each of them and use an abstract layer (API agnostic dev). Abstraction layers don't necessarily need to produce lowest common denominator results. Plus, AWS offers a wide range of strategies to ensure the availability of its infrastructure (Availability Zone, Regions, CDN etc.) You still wind up vulnerable to quirks of the single provider…

> Abstraction layers don't necessarily need to produce lowest common denominator results.

Except in the special case that you can build the functionality not provided by using other functionality that is, I think they do, since otherwise it's leaking. A leaky abstraction is often worse than none.

Re: Autoscaling on AWS

#23

Post by spulec »

mathrawka wrote: post #12

I have found that using a ubuntu base AMI and salt ( http://saltstack.com/community.html ) makes things much easier to manage. His suggestion of having node.js and nginx baked into the AMI is a big pain when you need to upgrade node.js or nginx (hello security update!). Using something like salt (or puppet or chef) would make this much easier. But in general, I agree with the article.

Agreed. We've had success using a very base ubuntu ami and then having the user data install chef and do a chef run.

This results in slower startups for new machines, but the flexibility seems to be worth it. Eventually we'll get to a point where the system will just build a new ami anytime a cookbook change is committed.

Re: Autoscaling on AWS

#24

Post by mnutt »

mqzaidi wrote: post #14
mathrawka wrote: post #12

I have found that using a ubuntu base AMI and salt ( http://saltstack.com/community.html ) makes things much easier to manage. His suggestion of having node.js and nginx baked into the AMI is a big pain when you need to upgrade node.js or nginx (hello security update!). Using something like salt (or puppet or chef) would make this much easier. But in general, I agree with the article.

We enable auto applying security updates in the AMI, and the first thing the provisioning script does is update packages. That said, I would definitely want to explore alternatives - not needing to update while provisioning will take a few seconds off the provisioning time.

What if you rebuilt the AMI every time you deployed?

Re: Autoscaling on AWS

#25

Post by crb »

mnutt wrote: post #24
mqzaidi wrote: post #14

Earlier quoted context omitted.

We enable auto applying security updates in the AMI, and the first thing the provisioning script does is update packages. That said, I would definitely want to explore alternatives - not needing to update while provisioning will take a few seconds off the provisioning time.

What if you rebuilt the AMI every time you deployed?

That's what Netflix do: http://techblog.netflix.com/2013/03/ami-creation-with-aminat...

Re: Autoscaling on AWS

#28

Post by bdunbar »

dmourati wrote: post #27
bdunbar wrote: post #26

Earlier quoted context omitted.

How does your organization mount volumes found on another host?

S3/EBS at the moment. MogileFS/SMB in the past though mogile and S3 are more properly referred to as object stores.

Ah. I'm new to AWS, had not yet explored 'storage' options.

I guess I'd been thinking of AWS as being 'vmware' but more so, without thinking hard about how it could be different.

Re: Autoscaling on AWS

#29

Post by contingencies »

randallsquared wrote: post #22
contingencies wrote: post #17

Earlier quoted context omitted.

Once you have two providers, you are forced to only use the features offered by each of them and use an abstract layer (API agnostic dev). Abstraction layers don't necessarily need to produce lowest common denominator results. Plus, AWS offers a wide range of strategies to ensure the availability of its infrastructure (Availability Zone, Regions, CDN etc.) You still wind up vulnerable to quirks of the single provider…

> Abstraction layers don't necessarily need to produce lowest common denominator results. Except in the special case that you can build the functionality not provided by using other functionality that is, I think they do, since otherwise it's leaking. A leaky abstraction is often worse than none.

I think your assumption is that cloud providers with different features cannot be abstracted without making the whole abstraction 'leaky'.

While your perspective may hold for a traditional, rigid, single-layer, abstraction layer with the most simplistic, binary-level feature presence, it does not hold for better formed solutions. From https://en.wikipedia.org/wiki/Abstraction_layer: All problems in computer science can be solved by another level of indirection (David Wheeler).

Some real world differences between cloud providers: available hardware, available installation images (OS images), available bandwidth, available logical location on the internet, available physical location (legal jurisdiction, etc.), scale-out time, cost model.

How do you conceive of these differences, and then deploy arbitrary services to an arbitrary number of individual instances running unique combinations of cloud provider specific features on unique cloud-provider deployed OS images in parallel? There are various approaches, but it's not that hard to come up with a functional set of abstractions. Think about it.

Post reply on HN