Amazon EC2 down?

#1

Post by vini » Fri, Jun 15, 2012, 4:29 AM UTC

Re: Amazon EC2 down?

#2

Post by ftwinnovations » Fri, Jun 15, 2012, 4:45 AM UTC

Yup, very down. My site is down, and I've been attempting to reboot and recreate instances like a madman... Why didn't I just check HN first??

Re: Amazon EC2 down?

#3

Post by [deleted] » Fri, Jun 15, 2012, 4:50 AM UTC

[deleted]

Re: Amazon EC2 down?

#4

Post by harryh » Fri, Jun 15, 2012, 4:51 AM UTC

FWIW as of approx 0450 UTC we're starting to see various instances that had become unresponsive return back to service.

Re: Amazon EC2 down?

#5

Post by derekclapham » Fri, Jun 15, 2012, 4:51 AM UTC

N. Virginia in my experience is by far the least reliable region on EC2/EBS... Fortunately our app servers are across 2 zone in the region... but our db server is just a lone master... Our slave is down... Very nervous.

Re: Amazon EC2 down?

#6

Post by csmcdermott » Fri, Jun 15, 2012, 4:52 AM UTC

Coming back up now...

Re: Amazon EC2 down?

#7

Post by WALoeIII » Fri, Jun 15, 2012, 5:00 AM UTC

derekclapham wrote: post #5

N. Virginia in my experience is by far the least reliable region on EC2/EBS... Fortunately our app servers are across 2 zone in the region... but our db server is just a lone master... Our slave is down... Very nervous.

I run a master with a "hot" master each in a different AZ and slaves of each in their respective AZs for days like today. Expensive, but makes it easy to sleep at night.

The slaves have their EBS disks snapshotted every 30 minutes, the master every 24 hours.

Re: Amazon EC2 down?

#8

Post by elq » Fri, Jun 15, 2012, 5:00 AM UTC

power outage in one of their AZ's. Only after lots of machines died did the generator kick on. Lovely.

Re: Amazon EC2 down?

#9

Post by jacobian » Fri, Jun 15, 2012, 5:02 AM UTC

elq wrote: post #8

power outage in one of their AZ's. Only after lots of machines died did the generator kick on. Lovely.

Do you have a source for this?

Re: Amazon EC2 down?

#10

Post by derekclapham » Fri, Jun 15, 2012, 5:02 AM UTC

WALoeIII wrote: post #7

derekclapham wrote: post #5

N. Virginia in my experience is by far the least reliable region on EC2/EBS... Fortunately our app servers are across 2 zone in the region... but our db server is just a lone master... Our slave is down... Very nervous.

I run a master with a "hot" master each in a different AZ and slaves of each in their respective AZs for days like today. Expensive, but makes it easy to sleep at night. The slaves have their EBS disks snapshotted every 30 minutes, the master every 24 hours.

Yeah... I snapshot our slaves every 30 minutes.. When you say "hot" master what exactly do you mean?