Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon EC2 currently down. Affecting Heroku, Reddit, Others

status.aws.amazon.com

Post reply on HN
Print view View original

301–310 of 311 posts

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#301

Post by acdha »

Supreme wrote: post #291
acdha wrote: post #219

Earlier quoted context omitted.

It's bizarre the way "cloud" makes so many people think disks never fail, networks are perfect and data centers always run smoothly. Now we'll get the backlash blog posts from people ditching the cloud – and I'm jousting waiting for the inevitable rebound outages when they learn that high availability requires geographic redundancy either way.

You find it bizarre that the cloud providers' marketing strategy has worked? I find that bizarre!

That's just silly given how much of Amazon's documentation strongly encourages you to use multiple AZs and regions for reliability. I've included a sampling of their whitepapers below; this is also what their salespeople tell you and what you have to click past any time you provision an RDS instance, ELB, etc.

http://media.amazonwebservices.com/AWS_Cloud_Best_Practices....

“Be a pessimist when designing architectures in the cloud”

http://media.amazonwebservices.com/AWS_Web_Hosting_Best_Prac... “As the AWS web hosting architecture diagram in this paper shows, we recommend that you deploy EC2 hosts across multiple Availability Zones to make your web application more fault-tolerant.”

http://media.amazonwebservices.com/AWS_Operational_Checklist...

“We have deployed critical components of our applications across multiple availability zones, are appropriately replicating data between zones, and have tested how failure within these components affects application availability”

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#302

Post by acdha »

toomuchtodo wrote: post #252
darrencauthon wrote: post #220

Earlier quoted context omitted.

There's not much sweating to do, as it always comes up relatively quickly.

How long was Amazon AWS "degraded" today?

2 minutes if you checked the "multi-AZ" box on your RDS instances or ELBs.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#303

Post by EternalLight »

xenophanes wrote: post #280
Angostura wrote: post #244

Earlier quoted context omitted.

Let me explain to you why it is a funny joke. It is funny because it involves Amazon undertaking massive technical measures, with huge reputational damage in order to try to kill a story which is primarily not spreading via Amazon-hosted sites anyway. It's akin to a man with athlete's foot deciding to remedy it by discharging a shotgun into his leg.

It's easy to understand shooting a leg with a shotgun. That's a simple thing. The Amazon thing in question is far more complicated, and far harder to understand. Thinking they are "akin" is a mistake. It shows you're thinking about it wrong and failing to recognize how completely different they are. One isn't going to confuse anyone or be misunderstood, the other will confuse most of the population and be misundersto…

Casually joking about impossible, evil conspiracies by big business on the other hand is something completely different and also funny.

No one will believe its related and its certainly not slander to joke about it. Also you might want to leave the political opinions out of hacker news... there is no 50% of US who dislikes those things, they only have different ideas about how to support it.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#304

Post by Supreme »

acdha wrote: post #301
Supreme wrote: post #291

Earlier quoted context omitted.

You find it bizarre that the cloud providers' marketing strategy has worked? I find that bizarre!

That's just silly given how much of Amazon's documentation strongly encourages you to use multiple AZs and regions for reliability. I've included a sampling of their whitepapers below; this is also what their salespeople tell you and what you have to click past any time you provision an RDS instance, ELB, etc. http://media.amazonwebservices.com/AWS_Cloud_Best_Practices.... “Be a pessimist when designing architectures…

Fair enough, I stand corrected. The problem is in the culture surrounding cloud services, not with the providers themselves.

So it seems that the only real benefit to utilising cloud services is to make scaling up easier and save money.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#306

Post by davidjnelson »

I have a few questions about HA AWS deployments. Anyone know the answers?

It seems obvious that multi region for HA is ideal, but the reference architecture for web apps on the aws architecture page only uses multi AZ for HA. How do you bridge that disconnect? Route53 with super low ttl? Wouldn’t route53 then be a single POF?

Also, is dynamodb multi region? If you have to use MySQL, how do you replicate cross region in a time and cost efficient manner? Isn’t S3 also only multi az, is there even a way to make s3 cross region? Or do you just “hope” dynamodb, route53, and s3 don’t fail?

Thanks, Best, David

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#307

Post by cstejerean »

vidarh wrote: post #292
mrkurt wrote: post #258

Earlier quoted context omitted.

Old school colo/dedicated servers/etc. There's something delightfully simple about only having to deal with "standard" hardware failures.

Not to mention unless you have very unusual traffic patterns (spin up lots of servers for short periods of time), colo/dedicated servers will usually be vastly cheaper than EC2, especially because with a little bit of thought you can get servers that are substantially better fit for your use. E.g. I'm currently about to install a new 2U chassis in one of our racks. It holds 4 independent servers each with with dual 6…

If disk performance is important you can also take a look at the High IO instances, which give you 2x 1TB SSDs, 60GB of RAM and 35 ECUs across 16 virtual cores. At 24x7 for 3 years you end up with ~$656/mo per instance, plus whatever you would need for bandwidth. By the time you fill up an entire rack it still ends up being slightly more expensive than your amortized 2U cost, but you also don't need to scale it up in 2U increments.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#308

Post by acdha »

Supreme wrote: post #304
acdha wrote: post #301

Earlier quoted context omitted.

That's just silly given how much of Amazon's documentation strongly encourages you to use multiple AZs and regions for reliability. I've included a sampling of their whitepapers below; this is also what their salespeople tell you and what you have to click past any time you provision an RDS instance, ELB, etc. http://media.amazonwebservices.com/AWS_Cloud_Best_Practices.... “Be a pessimist when designing architectures…

Fair enough, I stand corrected. The problem is in the culture surrounding cloud services, not with the providers themselves. So it seems that the only real benefit to utilising cloud services is to make scaling up easier and save money.

I think the main problem is that "the cloud"'s primary audience are developers, not sysadmins. Many developers simply don't appreciate that what you're getting is a [much] easier path to automating your server provisioning and management but you're still in exactly the same position as before regarding any bit of infrastructure's ability to fail at the least convenient moment.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#309

Post by strlen »

toomuchtodo wrote: post #267
strlen wrote: post #254

Earlier quoted context omitted.

As a start, be present in multiple EC2 availability zones (not just US-east-2, basically) and regions (this is harder). Cross-region presence needn't be active-active, just a few read-only database slaves and some machines to handle SSL termination ("points of presence") for your customers on the east coast. Perform regular "fire drills" where you actually fail over live traffic and primary databases from one AZ/one…

We run gear in multiple physical locations, but both the application and data is stored/backed in S3. This allows us the redundancy of S3 without the cost and fragile nature of EC2.

Indeed, there are many ways to complement physical colocation with AWS.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#310

Post by benwerd »

jedberg wrote: post #123
benwerd wrote: post #30

This would be a great time to post a guide to architecting systems for failover using AWS. Anyone got a great guide?

We (Netflix) have done a bunch of presentations on it which are on our slideshare page and across the internet. After this issue is over I can give a longer answer. In short, we've just evacuated the affected zone and are mostly recovered.

I'd love to hear that longer answer, fwiw!
Post reply on HN