AWS Tips I Wish I'd Known Before I Started • Hacker News Board

Re: AWS Tips I Wish I'd Known Before I Started

#61

Post by pnathan » Mon, Feb 03, 2014, 9:48 PM UTC

freerobby wrote: post #55

pavel_lishin wrote: post #37

Earlier quoted context omitted.

Based on the article, it seems it's there to make sure that you're automating everything, instead of logging in to do that one little thing by hand.

Thanks. Does anybody else here agree with this mentality? This seems a major mispractice to me. I've worked at companies with as few as two people to as many as 50,000 people. None of them have had production systems that are entirely self-maintaining. Most startups are better off being pragmatic than investing man-years of time handling rare error cases like what to do if you get an S3 upload error while nearly out…

I agree - it seems off to me. Sometimes you really want to diagnose your problems manually.

I'm also wondering how command and control is maintained without SSH access. Is there some kind of autoupdating service polling a master configuration management server (i.e., puppet's puppetmaster)?

I can appreciate that ensuring that a typical deploy doesn't require hand-twiddling. That makes sense, lots of it. But not disabling SSH.

Re: AWS Tips I Wish I'd Known Before I Started

#62

Post by richadams » Mon, Feb 03, 2014, 9:49 PM UTC

mbreese wrote: post #11

I'd also add to the list - make sure that AWS is right for your workload. If you don't have an elastic workload and are keeping all of your servers online 24/7, then you should investigate dedicated hardware from another provider. AWS really only makes sense ($$) when you can take advantage of the ability to spin up and spin down your instances as needed.

Absolutely, great point! AWS isn't for everyone, and there can be lots of cases where it's cheaper to use dedicated hardware. Shop around before jumping in. I've added this as a new tip at the end of the article (crediting you of course). Thanks!

Re: AWS Tips I Wish I'd Known Before I Started

#63

Post by Consultant32452 » Mon, Feb 03, 2014, 9:54 PM UTC

tedivm wrote: post #28

tyw wrote: post #12

Earlier quoted context omitted.

also the outbound bandwidth cost of S3 is very high. it would cost us several times what we're paying for s3+cloudfront to serve our content straight from s3.

Even cloudfront is ridiculously overpriced for a CDN. If you're pushing anything close to real bandwidth you could do a lot better elsewhere.

If you're pushing anything close to real bandwidth you're probably getting an individually negotiated price that is not public.

Re: AWS Tips I Wish I'd Known Before I Started

#64

Post by richadams » Mon, Feb 03, 2014, 9:54 PM UTC

Judson wrote: post #48

One thing the article mentions is terminating SSL on your ELB. If you want more control over your SSL setup AND want to get remote IP information (e.g. X-Forwarded-For) ELB now supports PROXY protocol. I wrote a little introduction on how to set it up[0]. They haven't promoted it very much, but it is quite useful. [0]: http://jud.me/post/65621015920/hardened-ssl-ciphers-using-aw...

Great post, I had no idea you could do this with ELB. I've added your link to the additional reading list in my post, thanks for sharing!

Re: AWS Tips I Wish I'd Known Before I Started

#65

Post by banachtarski » Mon, Feb 03, 2014, 10:02 PM UTC

AaronBBrown wrote: post #51

banachtarski wrote: post #46

Earlier quoted context omitted.

You speak as though ops and automation is too mysterious for a startup to handle. There are so many tools and frameworks that do what AWS does that it's easy to acquire that expertise. And who says you always need a new server to iterate quickly? I moved my last company completely off AWS and it was proven to be a great decision across a number of dimensions.

Can you list some of those tools and provide an estimate of how long they take to configure and how much day-to-day support they require?

Sure. Pop in opscode chef. Took me a weekend to write the basic framework, 3 weeks to make it solid. The 3 weeks more than paid for itself and hosting the config servers with them is a couple hundred a month. I could've hosted it myself too. This includes support for things like a load balancer, heterogeneous nodes (db, app, cache, chat, etc).

Ansible, puppet, sprinkler, and the like would take a similar amount of time to configure.

Re: AWS Tips I Wish I'd Known Before I Started

#66

Post by richadams » Mon, Feb 03, 2014, 10:02 PM UTC

mslot wrote: post #38

Be very careful with assigning IAM roles to EC2 instances. Many web applications have some kind of implicit proxying, e.g. a function to download an image from a user-defined URL. You might have remembered to block 127.0.0.*, but did you remember 169.254.169.254? Are you aware why 169.254.169.254 is relevant to IAM roles? Did you consider hostnames pointed to to 169.254.169.254? Did you consider that your HTTP client…

Great point with regards to IAM roles. The applications I've worked on don't download things from user-defined URLs, so this never even occurred to me.

Is the purpose of blocking 169.254.169.254 important because it could potentially give users access to the instance metadata service for your instance? I'd be interested to hear more information on securing EC2 with regards IAM roles, you seem to have lots of experience in that area.

The disabling SSH tip wasn't really about security (I agree that it has negligible security benefit), it's more about quickly highlighting parts of your infrastructure that aren't automated. It's often tempting to just quickly SSH in and fix this one little thing, and disabling it will force you to automate the fix instead.

The CDN info has been mentioned elsewhere too, lots of things I didn't know. I'll be updated the article soon to add all of the points that have been made. Thanks for the tips!

Re: AWS Tips I Wish I'd Known Before I Started

#67

Post by alexscript » Mon, Feb 03, 2014, 10:05 PM UTC

richadams wrote: post #49

paulgb wrote: post #9

Earlier quoted context omitted.

Also, if you change the first line of http://wblinks.com/css/style.css to @import url(http://fonts.googleapis.com/css?family=Droid+Sans:400,700); you should notice an improvement in the boldface font rendering. Great article, btw.

Thanks for the tip! CSS updated.

Since we're giving a little help here, you have two typos in this sentence.

> You can't change bucket names one you've creatd them, so you'd have to copy everything to a new bucket.

once and created

Re: AWS Tips I Wish I'd Known Before I Started

#68

Post by jcampbell1 » Mon, Feb 03, 2014, 10:11 PM UTC

micah_chatt wrote: post #58

mbreese wrote: post #45

Earlier quoted context omitted.

True, but outside of S3 and Route53, how much under the AWS umbrella is much use without using at least one EC2 instance? I can see a lot of benefit to using S3 without EC2, but after that, I'm not sure what else would be possible. Care to elaborate more? Can you use their queues and database tools w/o using EC2? (If you are using a VPC, maybe?)

"Can you use their queues and database tools w/o using EC2?" Yes, you very easily can. RDS is a simple setup, and so is SQS (queuing). The same goes for Dynamo and Redshift.

Are you really going tolerate 10+ ms of latency between a database, and an app server? I have never personally tried it, but it seems like it would be a disaster.

Re: AWS Tips I Wish I'd Known Before I Started

#69

Post by joevandyk » Mon, Feb 03, 2014, 10:12 PM UTC

bearbin wrote: post #33

AaronBBrown wrote: post #21

Earlier quoted context omitted.

As someone who has worked w/ AWS (and Rackspace) for several years with multiple startups... Unless they have dramatically improved their offering in the last couple years, an hour or two from "I need a new server" to delivery is 1) not an accurate timeframe for physical servers from Rackspace and 2) even if it was realistic, that's an eternity when you are trying to iterate quickly. I can have a new server in 30 sec…

DigitalOcean - Same or better speed, 1/6 or less of the price of ECC.

AWS is PCI compliant. http://aws.amazon.com/compliance/pci-dss-level-1-faqs/

DigitalOcean and most others are not.

AWS's security and API is lightyears ahead of everyone else.

Re: AWS Tips I Wish I'd Known Before I Started

#70

Post by crescentfresh » Mon, Feb 03, 2014, 10:13 PM UTC

j-kidd wrote: post #14

Good article, but I think it touches too little about persistence. The trade-off of EBS vs ephemeral storage, for example, is not mentioned at all. Getting your application server up and running is the easiest part in operation, whether you do it by hand via SSH, or automate and autoscale everything with ansible/chef/puppet/salt/whatever. Persistence is the hard part.

Good point. We're struggling to see the benefits of EBS for Cassandra that has its own replication strategy (ie data is not lost if an instance is lost), voiding the "only store temporary data on ephemeral stores" argument.