Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon EC2 and RDS in US-EAST zone down

status.aws.amazon.com

Post reply on HN
Print view View original

71–80 of 98 posts

Re: Amazon EC2 and RDS in US-EAST zone down

#71

Post by res0nat0r »

mikebo wrote: post #61
res0nat0r wrote: post #59

Earlier quoted context omitted.

Did/does your standby replica in another AZ have any instance notifications stating there is a failure? The outage report claims there were just EBS problems in only one AZ.

No, nothing unusual with our standby replica. It's not even clear if it was the standby or our primary that was in the affected AZ. Multi-AZ RDS does synchronous replication to the standby instance -- I'm guessing something broke in there. Hopefully AWS will update with a post mortem as they usually do. Lots of frustrated MultiAZ RDS customers on their forums.

Yeah unfortunately it looks to be an EBS problem and if your underlying EBS volume housing your primary DB instance takes a dump then that is unfortunately going to cause replication to fall over too

Re: Amazon EC2 and RDS in US-EAST zone down

#72

Post by gouranga »

TazeTSchnitzel wrote: post #64
gouranga wrote: post #36

Earlier quoted context omitted.

That sucks badly. Similar thing happened to me a while ago with a vendor. When your management team summons you to ask why the hell their site is down, you can't point fingers at the vendor if their marketing literature says it doesn't go down. Sticky situation.

Can't you tell management that it isn't as reliable as they claim?

I did. Unfortunately in the financial services industry, believing it means taking responsibility for it.

Re: Amazon EC2 and RDS in US-EAST zone down

#73

Post by oasisbob »

zedwill wrote: post #25

Interesting enough not only the EBS is down, but ELB can not register instances even if there are not EBS based and completely operational. I have some live instances running without EBS disks that I can not place behind the ELB as it is not working.

I have some live instances running without EBS disks that I can not place behind the ELB as it is not working.

ELBs are sometimes EBS backed.

Re: Amazon EC2 and RDS in US-EAST zone down

#74

Post by bmelton »

tolos wrote: post #50

Every time (two out of two), by the time I click on "X is down" link, the service/website is working again. Surely there is a better platform for alerting about outages than ycombinator?

I was down for approximately three hours this morning. I don't know when this submission was posted, but I made one shortly after discovering the outage myself.

Either way, if you're using RDS, even if this didn't affect you, it's discussion-worthy. I was affected, and we're building a not-yet-launched product that allows us the time to consider "Is Amazon really where we want to be?". The more failure I'm aware of, the more informed that decision is.

Re: Amazon EC2 and RDS in US-EAST zone down

#75

Post by pjscott »

EC2 comes with a free Chaos Monkey service. It's called EC2.

I know, they're trying to make it reliable and they've got a bunch of very hard problems to solve. That doesn't change the fact that sometimes some of my servers just permanently stop responding to pings until you stop-start them, or get crazy-slow I/O, or get hit by these once-in-a-while-and-always-at-night outages.

It's great when you suddenly need a hundred more servers, though.

Re: Amazon EC2 and RDS in US-EAST zone down

#76

Post by mikebo »

res0nat0r wrote: post #71
mikebo wrote: post #61

Earlier quoted context omitted.

No, nothing unusual with our standby replica. It's not even clear if it was the standby or our primary that was in the affected AZ. Multi-AZ RDS does synchronous replication to the standby instance -- I'm guessing something broke in there. Hopefully AWS will update with a post mortem as they usually do. Lots of frustrated MultiAZ RDS customers on their forums.

Yeah unfortunately it looks to be an EBS problem and if your underlying EBS volume housing your primary DB instance takes a dump then that is unfortunately going to cause replication to fall over too

Multi-AZ RDS deployment is supposed to protect you from that though. That's why it's 2x the price. We should have failed over to a different AZ w/o EBS issues.

Re: Amazon EC2 and RDS in US-EAST zone down

#77

Post by NoPiece »

sausagefeet wrote: post #56
pearle wrote: post #53

Anyone have any details on why us-east-1 seems to be less reliable than the other regions? Is it the oldest?

I'm under the impression it's the most used.

It probably is the most used, being a cheaper alternative to us-west, but are you suggesting it fails more because it is used more? It does seem that the big AWS outages (in the us) have been concentrated in us-east. I have wondered if it just because us-east is newer so they haven't had has much time to work things out, or that the us-west team is a little better?

edit: btw, I am not dismissing "used more" as a valid theory. More use = more hardware = more complexity which could lead to more failures.

Re: Amazon EC2 and RDS in US-EAST zone down

#78

Post by pjscott »

tolos wrote: post #50

Every time (two out of two), by the time I click on "X is down" link, the service/website is working again. Surely there is a better platform for alerting about outages than ycombinator?

Pingdom does a good job of it, if you point it at a public-facing web site you particularly care about. I'm not affiliated with them; I've just been woken up by them.

Re: Amazon EC2 and RDS in US-EAST zone down

#79

Post by res0nat0r »

mikebo wrote: post #76
res0nat0r wrote: post #71

Earlier quoted context omitted.

Yeah unfortunately it looks to be an EBS problem and if your underlying EBS volume housing your primary DB instance takes a dump then that is unfortunately going to cause replication to fall over too

Multi-AZ RDS deployment is supposed to protect you from that though. That's why it's 2x the price. We should have failed over to a different AZ w/o EBS issues.

If your source EBS volume is horked then you aren't going to be replicating any data to your backup host while the EBS volume is messed up (since your source data is unavailable). EBS volumes also don't cross/failover between AZ boundaries.

Maybe there was something bad with your replication server before the outage? It's hard to guess without knowing exactly what was happening at the time...

Re: Amazon EC2 and RDS in US-EAST zone down

#80

Post by rdl »

malachismith wrote: post #46
rdl wrote: post #38

I'm curious why no public paas is multiple AWS region.

1) because AWS East is so much cheaper (and none of us like spending money) 2) AppFog actually is multi region (and multi IaaS as well)

I'd tolerate multi-AZ as a baseline.

Thanks for AppFog -- I hadn't heard of them, but will check them out.

Post reply on HN