Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon EC2 currently down. Affecting Heroku, Reddit, Others

status.aws.amazon.com

Post reply on HN
Print view View original

91–100 of 311 posts

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#91

Post by wyck »

caseysoftware wrote: post #57
caseysoftware wrote: post #46

Earlier quoted context omitted.

We (Twilio) have released a number of articles & presentations in this area: http://www.slideshare.net/twilio/highavailability-infrastruc... http://www.twilio.com/engineering/2011/04/22/why-twilio-wasn... It's strategy as opposed to how-to but the principles apply.

And Netflix's Simian Army is awesome: http://techblog.netflix.com/2011/07/netflix-simian-army.html They've even released the "Chaos Monkey" open source: http://techblog.netflix.com/2012/07/chaos-monkey-released-in...

Yet netflix is currently down, so maybe there is a problem with the chimp army?

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#92

Post by blaze33 »

Hosted on heroku with a hosted heroku postgres dev instance, I observed a drop in the db response time :

- from a consistent average of 10ms over the last week

- to a new consistent average of ~2.5ms

beetwen 17:31 and 17:35 UTC. AWS started to report the current issue at 17:38 UTC. My app then experienced some intermittent issues (reported by newrelic pinging it every 30s). Don't know if it's related, could it be some sort of hw upgrade that went wrong ?

I did a push affecting my most used queries but that was one and a half hour sooner, at 15:56 UTC, so probably unrelated.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#93

Post by jasonkester »

For added fun, their EC2 console is down. I got this for a while:

  Http/1.1 Service Unavailable
... then an empty console saying "loading" for the last 20 minutes. Then recently it upgraded to saying "Request limit exceeded." in place of the loading message (because hey, I'd refreshed the page four times over the course of 20 minutes).

On the upside, their status page shows all green lights.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#94

Post by engtech »

btilly wrote: post #85

As https://twitter.com/DEVOPS_BORAT says, At conference you can able tell cloud devops by they are always leave dinner for respond to pager. Also, What is happen in cloud is stay in cloud because nobody can able reproduce outside of cloud. (And many other relevant quotes.)

so true: "In devops is turtle all way down but at bottom is perl script. "

https://twitter.com/DEVOPS_BORAT/status/248770195580125185

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#95

Post by caseysoftware »

caseysoftware wrote: post #57
caseysoftware wrote: post #46

Earlier quoted context omitted.

We (Twilio) have released a number of articles & presentations in this area: http://www.slideshare.net/twilio/highavailability-infrastruc... http://www.twilio.com/engineering/2011/04/22/why-twilio-wasn... It's strategy as opposed to how-to but the principles apply.

And Netflix's Simian Army is awesome: http://techblog.netflix.com/2011/07/netflix-simian-army.html They've even released the "Chaos Monkey" open source: http://techblog.netflix.com/2012/07/chaos-monkey-released-in...

I'm watching Netflix from Austin, so it's not entirely down in the US.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#96

Post by iamdave »

mey wrote: post #50
res0nat0r wrote: post #21

Earlier quoted context omitted.

Many large sites which aren't properly architecting their infrastructure to deal with the commodity nature of the cloud.

http://downrightnow.com/netflix If netflix is down, then it's something most companies who know how to design fail over can't cope with.

Netflix seems to be operating just fine for me in the southwest USA.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#98

Post by hornbaker »

jasonkester wrote: post #93

For added fun, their EC2 console is down. I got this for a while: Http/1.1 Service Unavailable ... then an empty console saying "loading" for the last 20 minutes. Then recently it upgraded to saying "Request limit exceeded." in place of the loading message (because hey, I'd refreshed the page four times over the course of 20 minutes). On the upside, their status page shows all green lights.

They've acknowledged that at http://status.aws.amazon.com/ for awhile now with a tiny "i" status icon (I can't load my instances pane):

12:07 PM PDT We are experiencing elevated error rates with the EC2 Management Console.

Post reply on HN