I reacted quickly enough to get a new instance spun up and added it to my site's load balancer, but the load balancer is failing to recognize the new instance and send traffic to it... yay. Console is totally unresponsive on the load balancer's "instances" tab. If I point my browser at the ec2 public DNS of the new instance it seems to be running just fine. So much for the load balancer being helpful.
Amazon EC2 currently down. Affecting Heroku, Reddit, Others
61–70 of 311 posts
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#62Does anyone know if this really is just one AZ? Seems like an awful lot of larger sites are down. I'd expect at least some of them to be multi-AZ by now.
Multiple AZs here.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#63At least HN isn't down! :D
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#64back up :)
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#65pg, or someone else from HN... Could you please edit this title for accuracy? Maybe, "Poorly designed sites taken out because of problems in one Amazon availability zone."
That's editorializing.
For many sites, a single server in a single zone (e.g., a non redundant server, an instance, a slice, a VM, whatever) is the right decision for ROI.
For many sites, the money spent on redundancy could be better spent on, say, Google Adwords, until they're big enough that a couple hours downtime has irreplaceable costs higher than the added costs of redundancy (dev, hosting, admin) for a year.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#66I suppose this explains why Airbnb is down now, as well: http://aws.amazon.com/solutions/case-studies/airbnb/ I was just in the middle of booking a stay in a Palo Alto startup embassy for this week, too!
seems they should have a back up
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#67My dashboard says "Degraded EBS performance in a single Availability Zone". It then lists each of the five zones as "Availability zone is operating normally." http://cl.ly/image/202F3B0I371g
I was seeing that too, but it looks like Amazon has now updated the availability zone status. When I run ec2-describe-availability-zones from the command line, it's telling me that us-east-1b is impaired. (Availability zone mapping is different for each account, so my 1b may be some other availability zone for you.)
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#68The N. Virginia datacenter has been historically unreliable. I moved my personal projects to the West Coast (Oregon and N. California) and I have seen no significant issues in the past year. N. Virginia is both cheaper and closer to the center of mass of the developed world. I'm surprised Amazon hasn't managed to make it more reliable.
One thing we discovered this morning: it appears the AWS console itself is hosted in N Virginia. This means that if you were trying to make changes to your EC2 instances in the West using the GUI, you couldn't, even though the instances themselves were unaffected.
shouldn't amazon themselves have architected their own app to be able to move around?
I get tired of the snipes from people that "well, you're doing it wrong", as if this is trivial stuff. But if Amazon themselves aren't even making their AWS console redundant between locations, how easy/straightforward is it for anyone else?
To what extent is this just "the cobbler's kids have no shoes?"
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#69Earlier quoted context omitted.
Many large sites which aren't properly architecting their infrastructure to deal with the commodity nature of the cloud.
http://downrightnow.com/netflix If netflix is down, then it's something most companies who know how to design fail over can't cope with.
The last Netflix post mortem mentioned they had a bug in their configuration where they kept sending traffic to already down ELB instances, which was the cause of the last outage for them if I remember correctly.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#70Our app is down because it's hosted on Heroku and it's frustrating because it seems like N Virginia is the least reliable Amazon datacenter. Every year it seems to go down a couple times for at least a couple hours.
Heroku should offer a choice between N Virginia and Oregon hosting (I think they're almost comparable in price nowadays). That way people who want more uptime/reliability can choose Oregon. Sure it will be further from Europe (but then it will be closer to Asia) and people can make that choice on their own.
But basing an entire hosting service on N Virginia doesn't make sense anymore, considering the history of major downtime in that region.