power outage in one of their AZ's. Only after lots of machines died did the generator kick on. Lovely.
Do you have a source for this?
'tis what I heard on an outage call...
Amazon EC2 down?
11–20 of 64 posts
Re: Amazon EC2 down?
#12We learnt our lesson the hard way after the great AWScalypse of Apr 2011.
The lesson: Use n>1 hosting companies (even if one of them promises a-z-multiregion-distributed-fault-tolerant-back-up)
Re: Amazon EC2 down?
#13Earlier quoted context omitted.
I run a master with a "hot" master each in a different AZ and slaves of each in their respective AZs for days like today. Expensive, but makes it easy to sleep at night. The slaves have their EBS disks snapshotted every 30 minutes, the master every 24 hours.
Yeah... I snapshot our slaves every 30 minutes.. When you say "hot" master what exactly do you mean?
A "full-spec" machine, another X-Large with 4 EBS volumes that I can fail over to. Its in circular replication with the other "active" master (only one is receiving writes at a time). These instances are only snapshotted once a day to keep them as fast as possible.
Re: Amazon EC2 down?
#14power outage in one of their AZ's. Only after lots of machines died did the generator kick on. Lovely.
Are you sure? I have a graph of one of my Resque boxes taking 30Gbit/s of inbound traffic. Looks like a DDoS attack to me.
Re: Amazon EC2 down?
#15N. Virginia in my experience is by far the least reliable region on EC2/EBS... Fortunately our app servers are across 2 zone in the region... but our db server is just a lone master... Our slave is down... Very nervous.
N. Virginia is by far the most used region. There's more stuff to fail and any failure will affect more people. I don't think there's anything inherently less reliable about the geographical location of the building.
Re: Amazon EC2 down?
#16power outage in one of their AZ's. Only after lots of machines died did the generator kick on. Lovely.
Are you sure? I have a graph of one of my Resque boxes taking 30Gbit/s of inbound traffic. Looks like a DDoS attack to me.
The update I heard was (essentially) 'Another update from Amazon: Looks like it was a power issue for one facility that services a particular AZ in us-east-1, flipped to generator, now back on power and in recovery mode.'
Re: Amazon EC2 down?
#17Earlier quoted context omitted.
I run a master with a "hot" master each in a different AZ and slaves of each in their respective AZs for days like today. Expensive, but makes it easy to sleep at night. The slaves have their EBS disks snapshotted every 30 minutes, the master every 24 hours.
Yeah... I snapshot our slaves every 30 minutes.. When you say "hot" master what exactly do you mean?
[deleted]
Re: Amazon EC2 down?
#18Hootsuite just reported that it is offline - I'm unsure if it is related.
In other news, for once Reddit is working.
Re: Amazon EC2 down?
#19We learnt our lesson the hard way after the great AWScalypse of Apr 2011. The lesson: Use n>1 hosting companies (even if one of them promises a-z-multiregion-distributed-fault-tolerant-back-up)
how are you quickly failing over from one to another?
Re: Amazon EC2 down?
#20Our instances still down.
The AWS service health says:
9:27 PM PDT We continue to investigate this issue. We can confirm that there is both impact to volumes and instances in a single AZ in US-EAST-1 Region. We are also experiencing increased error rates and latencies on the EC2 APIs in the US-EAST-1 Region.