This would be a great time to post a guide to architecting systems for failover using AWS. Anyone got a great guide?
PagerDuty's CTO gave a talk that mentions it: http://blog.pagerduty.com/2012/10/ensuring-the-call-goes-out...
+1
Amazon EC2 currently down. Affecting Heroku, Reddit, Others
211–220 of 311 posts
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#212Convenient that we're too backward to use AWS. That means everyone can at least talk about it here when AWS is down.
I wish that DNS could just switch over to another availability zone when this happens. A datacenter with all the sql replicated. Sure you'd pay twice as much for EBS, but it could also double as a backup. As for the other resources at the availability zone, they are hardly utilized until they spring into action (EC2, etc.)
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#213All of our EC2 hosts appear to be functioning fine, but they can't connect to their RDS instance which renders our app useless. If you scroll down the page you'll also see that RDS instances are having connectivity issues. Not sure if it's related but for RDS users the impact is far worse. EDIT: We are also using multi-AZ RDS, so either Amazon's claims for multi-AZ are bs, or their claims that this is only impacting…
Because RDS is built on EBS, any slight issue with EBS manifests itself as a nastier issue for RDS.
Interestingly, EBS will never return an I/O error up to the attached OS, which is likely a good decision as most OSes choke on disk errors. What this means, however, is that if something even get slow within EBS (let alone stuck), applications that are dependent on it will suffer. Most of these applications (such as databases) have connection/response timeouts for their clients, so while EBS might just be running slowly, a service like RDS will throw up connection errors instead of waiting even a bit more.
You can imagine the cascading errors that might result from such a situation (instance looks dead, start failover...etc)
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#214Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#215Earlier quoted context omitted.
I think it's a case of "well optimized code which does as little as possible can handle an awful lot of users on a single beefy box".
Until it can't. Then five years later you get Spanner.
You can buy some seriously big boxes, and easily split off a lot of services onto multiple boxes. The big problem with the "single big box" strategy is being able to do upgrades -- I see hn go down frequently for 5-10 min at a time in the middle of the night, which I assume is upgrades/reboots.
The happy medium is probably splitting database (master/slave at least) and cdn (if needed) and some other services (AAA? logging?) out, and then having 2+ front end servers with load balancing for availability.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#216Maybe I'm being daft. Why does this sh*t never seem to affect www.amazon.com
Amazon doesn't run their site off the same set of EC2 servers and systems that everyone else does. They just used their experience to build it.
Amazon runs all of the Amazon.com web servers on EC2 hosts (and has since 2010: https://www.youtube.com/watch?feature=player_detailpage&...). They've just made sure that they have enough hosts in each of the other AZs to withstand an outage.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#217Earlier quoted context omitted.
Many large sites which aren't properly architecting their infrastructure to deal with the commodity nature of the cloud.
http://downrightnow.com/netflix If netflix is down, then it's something most companies who know how to design fail over can't cope with.
According to a site which doesn't document what its reports are based on. Given that Netflix worked for me during that period, I'm suspicious that downrightnow might be using EBS somewhere.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#218Earlier quoted context omitted.
To be fair, AWS downtime always make the news because they affect a lot of majors websites, but that doesn't mean an average sysadmin (or devops, whatever) would do better in term of uptime with his own bay and his toys.
SoftLayer that we use seems to be much more reliable than Amazon. At least more reliable than that particular Amazon's datacenter in Virginia.
I agree, been with SL for 3 years, never had an outage apart from a drive failure one time but it was fixed within an hour.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#219Convenient that we're too backward to use AWS. That means everyone can at least talk about it here when AWS is down.
Funny thing is, the last couple interviews I've had in Chicago and Silicon Valley, I actually get points when explaining caution is necessary when using AWS for production. A magic bullet it isn't.
It's bizarre the way "cloud" makes so many people think disks never fail, networks are perfect and data centers always run smoothly. Now we'll get the backlash blog posts from people ditching the cloud – and I'm jousting waiting for the inevitable rebound outages when they learn that high availability requires geographic redundancy either way.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#220Earlier quoted context omitted.
To be fair, AWS downtime always make the news because they affect a lot of majors websites, but that doesn't mean an average sysadmin (or devops, whatever) would do better in term of uptime with his own bay and his toys.
When our gear is down, we can actually get into the datacenter to fix it. What do you do when Amazon is down other than sweat?
There's not much sweating to do, as it always comes up relatively quickly.