Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: the good, the bad and the ugly

blog.awe.sm

Post reply on HN
Print view View original

81–88 of 88 posts

Re: AWS: the good, the bad and the ugly

#81

Post by seldo »

jwilliams wrote: post #40

Nice read, but I wish they had included the location (and AZs) that are in use. I've used Oregon, California and Virginia with different results. The comment around Ubuntu is interesting and I wish there was more detail there. We use mdadm to run RAID across multiple EBSes. mdadm is great, but has a kink that it will boot to a recovery console if there the volume is "degraded" (i.e. any failure). This is even if the…

We are primarily in us-east-1, as I mentioned in the post, with a skeleton set of DB slaves sitting in us-west as an emergency recovery if all of east-1 goes down.

In terms of AZs, were are distributed roughly evenly across all AZs in east-1.

Re: AWS: the good, the bad and the ugly

#82

Post by seldo »

ceejayoz wrote: post #28
seldo wrote: post #6

Earlier quoted context omitted.

We use Percona's XtraDB streaming backup to take backups of our smaller databases, and incremental backups of the larger DBs. We store them in a series of places: on a backup instance within east-1, on a second dedicated backup instance in west-1 (in case east-1 ever bites the dust completely, such as during hurricane Sandy), and then long-term archival on S3. S3 is good for smaller databases but for our biggest ones…

Any good resources for XtraDB's streaming backups? I'm looking to move in that direction early next year.

We are planning a follow-up blog post that goes into more technical detail about our backup and restore strategies; look for it in a week or two.

Re: AWS: the good, the bad and the ugly

#83

Post by seldo »

spartango wrote: post #62
daemon13 wrote: post #58

From alestic.com >> Both EBS boot and instance-store AMI ids are listed, but I recommend you start with EBS boot AMIs. Why two opposite recommendations from alestic.com [authority on AWS] and practitioners? Not a flame - I am planning my AWS deploy strategy and need to make a decision between these two approaches.

As a bit of background: a lot of the performance pain around EBS is the inconsistency in performance between IOs. Many times you'll get a nice fast op, but sometimes IOs will get slow suddenly, or even stuck. You can imagine that internally, this is EBS seeking on magnetic disks or getting blocked by a clogged network. Regardless, inconsistent performance is nearly as bad (or perhaps worse) than consistently bad perf…

As you say, the real trouble with EBS is when serving application data. We run a lot of databases, and EBS was just not good enough for them even when there were no system-wide issues.

Our problem with EBS as a boot device was that occasional system logs, etc. would go it, and when EBS was down the inability to reach the disk (the block-store abstraction problem) would lock up the whole OS.

Re: AWS: the good, the bad and the ugly

#84

Post by ceejayoz »

jiggy2011 wrote: post #80
ceejayoz wrote: post #78

Earlier quoted context omitted.

> I assume it runs a standard linux distro, therefor patches , firewalls, dependencies etc are still an issue surely? AWS's security groups pretty well cover you on the firewalls front (although some folks like to still run software ones on their instances to be doubly sure), but you're otherwise correct. For a single instance, you're not going to see significant differences. It's running large clusters of instances…

I mean assuming that you want to open SSH you are going to need to secure that either using keys or fail2ban. If you have a web server you might need mod_security (or does AWS reverse proxy everything?). If you want remote database access you need to configure that too. Does AWS provide easier tools for that stuff? Because you can't really cover it with a general firewall.

The only things exposed on our VPC are an Amazon ELB and a well-secured SSH bastion host (keys only, EC2 instances have password login disabled by default) for getting into the private cloud.

No reason you can't run stuff like mod_security, but that's not strictly a firewall, just Apache setup.

Re: AWS: the good, the bad and the ugly

#85

Post by patrickgzill »

ghshephard wrote: post #44
blantonl wrote: post #24

This is a great writeup and is completely on target for realistic deployments on AWS. We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post: * I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forwar…

I don't get the 100tb.com model. Checking out their website, I see: Intel E3-12303.2 GHz8GB2 x 1TB100 TB$201.15 1Gbit Dedicated Port So, for $200/month, they'll give me 100 Terabytes/bandwidth on this server. 250 megabit/second at 30 days in terabytes = 81 Terabytes. A decently peered/connected Pipe costs, at this volume, around $5-$7/megabit @95th, or $1250.month. So either: A) Their connectivity isn't hot. B) If yo…

Current bandwidth pricing: Level3 (quite good) about $3/Mbps, Cogent (not as good), 70 cents/Mbps. So $700 to $3000 for a 1Gbit pipe depending on quality, though usually they will charge you a bit more and you will have 1Gbps on a burstable 10Gbe connection.

However - most people don't use anywhere near this amount - really, they are selling you a server connected to a gigabit switch.

As soon as over-subscription kicks in (it does immediately, since you don't start using 250Mbps the day you have your server installed) their bandwidth costs go to a fraction of what you are paying monthly.

Re: AWS: the good, the bad and the ugly

#86

Post by ra »

dogas wrote: post #69
ra wrote: post #60

Earlier quoted context omitted.

Really interested to learn about how you use Chef to bring up and configure your ephemeral instances. Also, do you backup your databases anywhere other than on other ephemeral instances?

We still use EBS for database backups via ec2-consistent-snapshot. The slave DB that performs this is not production-facing. We have 2 separate chef recipes for our DBs: One is for a full-time slave. This recipe will set up the db to use the EBS volume. The other is for a slave that will be promoted to a master. In this case, we do a little extra legwork to do a bit-by-bit copy of a recent EBS snapshot, onto the ephe…

Thanks for replying. And yes I think you're ahead of the curve on this!

Re: AWS: the good, the bad and the ugly

#87

Post by zurn »

spartango wrote: post #64
zurn wrote: post #63

Earlier quoted context omitted.

Hardware and software have co-evolved, so that disks provide an illusion of error free operation until they throw in the towel and die. And they have consistent performance. This has worked OK so far. With network filesystems (eg. NFS) you can choose to return an I/O error to the application when you hit a timeout or a network error (the -o intr mount option). This is rarely used since applications aren't used to dea…

On one hand you're right: when block devices fail, they're pretty much gone--if you've ever tried to read from a bad sector you'll know this exactly, or worse still on SSDs, where the disk will fail to show up to the bus. That said, I'm not sure I agree with the idea that we got away with a lack of error handling because disks had consistent performance. Magnetic disks have always had incredibly inconsistent random I…

> Magnetic disks have always had incredibly inconsistent random IO performance

From a non-realtime app POV disk seek performance is pretty consistent: you get 5-25ms seek cies that center around 10ms. Especially In contrast to network backed where you get to contend with hiccups and contention with other users.

OS disk caching came about for a different reason.

Re: AWS: the good, the bad and the ugly

#88

Post by jaytaylor »

Zombieball wrote: post #66
jaytaylor wrote: post #43

Earlier quoted context omitted.

SNS is frequently a huge PITA to get working, often requiring SSH tunnels and the like.

Not sure I follow. Could you elaborate how SNS would require an SSH tunnel to work? I've never encountered this issue in environments I have worked in.

I looked into this and I was actually mistaken - I was referencing SES, not SNS. Apologies!
Post reply on HN