Amazon Web Services are down • Hacker News Board

Re: Amazon Web Services are down

#81

Post by crb » Thu, Apr 21, 2011, 10:57 AM UTC

StavrosK wrote: post #8

kylec wrote: post #5

I guess this is one Reddit outage that can't be blamed on poor scaling

Apparently most of their problems are caused by bad EBS writes/performance, or at least so they said a few weeks ago after some particularly bad downtime. It looks like EBS will randomly decide to switch to a few bps of performance from time to time. I would use Amazon for my startup, but these issues really make it hard to justify.

EC2 instances only have one network interface. The public IP address you have pointing to your instance is a DNAT done somewhere further up the chain.

If you get a large network load to your instance - say, a DDoS attack - you can find you no longer have enough network capacity to talk to your EBS disks.

This is what happened to Bitbucket in 2009: http://blog.bitbucket.org/2009/10/04/on-our-extended-downtim...

Re: Amazon Web Services are down

#82

Post by ig1 » Thu, Apr 21, 2011, 10:59 AM UTC

jedberg wrote: post #67

oomkiller wrote: post #58

They better start writing their explanation now. Multiple AZ's affected?

They better be cutting me a check too.

As an advertiser on Reddit, you guys better be cutting me a check too :)

(although my app is also dead right now so not as if the advertising would do me any good...)

Re: Amazon Web Services are down

#83

Post by 6ren » Thu, Apr 21, 2011, 11:00 AM UTC

jedberg wrote: post #34

mathrawka wrote: post #23

I think this is a good example of how the "cloud" is not a silver bullet to making your site always up. AWS provides a way to keep it up, but it is up to each developer to ensure that they are using AWS in a way to make sure their site can handle problems in one availability zone. I think we will see more of a focus from big users of AWS about focusing on how to create a redundant service using AWS. Or at least I hop…

This outage is affecting all AZ's in the East. So even a multizone setup wouldn't help for this one. Only a multiregion setup. This outage is a lot like having your entire datacenter lose power.

If "multi-region" means North America, Europe and Asia Pacific, doing so would also improve world-wide latency (e.g. here in Australia...).

Could you use this outage to justify switching to multi-region?

Re: Amazon Web Services are down

#84

Post by endgame » Thu, Apr 21, 2011, 11:00 AM UTC

taitems wrote: post #49

athesyn wrote: post #33

Earlier quoted context omitted.

My four-day weekend is already off to a bad start(UK here).

It could be worse/better, you could have an Australian 5-day weekend.

To clarify, this Monday is ANZAC Day, a commemorative holiday for troops who fought for Australia. Because that's also Easter Monday, the ANZAC Day holiday is moved to Tuesday, despite commemorative services being held on Monday.

Re: Amazon Web Services are down

#85

Post by jwr » Thu, Apr 21, 2011, 11:02 AM UTC

It's really mostly EBS failures, so the title is overly dramatic. And EBS has been known to have issues.

Re: Amazon Web Services are down

#86

Post by thirtysixred » Thu, Apr 21, 2011, 11:03 AM UTC

jedberg wrote: post #15

kylec wrote: post #5

I guess this is one Reddit outage that can't be blamed on poor scaling

Thankfully, no. :) But yeah, right now we're shutting everything down to try and avoid possible data corruption. Once they restore service, hopefully we'll be able to come back quickly.

SHUT. DOWN. EVERYTHING.

Good plan though. :p

Re: Amazon Web Services are down

#87

Post by mootothemax » Thu, Apr 21, 2011, 11:03 AM UTC

hendi_ wrote: post #16

Yay for relying on the cloud \o/

What do you do when a building contractor cuts through your datacentre's link? For example, this happened in London a while back.

My point is: things go wrong. Blaming this on the cloud is silly.

Re: Amazon Web Services are down

#88

Post by chrislomax » Thu, Apr 21, 2011, 11:06 AM UTC

chrislomax wrote: post #52

jedberg wrote: post #25

Earlier quoted context omitted.

Believe me, the last thing I want is to be up at 3am working on this. I'd much rather be sleeping and letting you not work.

Are you part of the team resolving it?

I was looking at EC2 until this!

I thought you could cluster your instances across many regions and replicate blah blah blah and change your elastic ip addresses in instances like this?

Is this a case that it's not being utilised or does that system not work?

I appreciate you are busy right now so I'm not expecting a reply any time soon.

Re: Amazon Web Services are down

#89

Post by mike-cardwell » Thu, Apr 21, 2011, 11:06 AM UTC

samratjp wrote: post #26

yuvadam wrote: post #2

Current status: bad things are happening in the North Virginia datacenter. EC2, EBS and RDS are all down on US-east-1. Edit: Heroku, Foursquare, Quora and Reddit are all experiencing subsequent issues.

http://status.heroku.com/ for those on Heroku

Is it just me, or is their status page down?

Re: Amazon Web Services are down

#90

Post by retube » Thu, Apr 21, 2011, 11:06 AM UTC

thirtysixred wrote: post #86

jedberg wrote: post #15

Earlier quoted context omitted.

Thankfully, no. :) But yeah, right now we're shutting everything down to try and avoid possible data corruption. Once they restore service, hopefully we'll be able to come back quickly.

SHUT. DOWN. EVERYTHING. Good plan though. :p

Reminds me of that scene in Jurassic Park