The AWS status page[1] is showing problems for EC2 East as of a few minutes ago. This might be is a more widespread issue. EDIT: Various non-Heroku EC2-East-based sites (e.g. Quora) seem to be down as well, lending more evidence to this being an EC2/EBS outage. 1: http://status.aws.amazon.com/
Out of curiosity do we know if http://status.aws.amazon.com/ is hosted on AWS?
Tell HN: Heroku is Down (update: recovering as of 10PM PST)
61–70 of 116 posts
Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)
#62Earlier quoted context omitted.
I just got up (it's morning in Israel), and a client of mine in the US with a major, mission-critical application was screaming (rightly so) that things are down. We're already looking into alternatives -- perhaps not leaving Heroku altogether, but certainly not depending on them 100 percent. There's no way that we can entrust the business to something that can just catastrophically fail at any moment. I've been runn…
Dont overreact... heroku is using AWS which is down for many other sites right now other than heroku. This is an EBS issue that is occurring.
Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)
#63The AWS status page[1] is showing problems for EC2 East as of a few minutes ago. This might be is a more widespread issue. EDIT: Various non-Heroku EC2-East-based sites (e.g. Quora) seem to be down as well, lending more evidence to this being an EC2/EBS outage. 1: http://status.aws.amazon.com/
Half the startup world is offline right now, yet the AWS status page is all greens with one "info" notice about EBS (which I doubt is the issue here). I'm glad to have moved off a fully AWS stack back to my own servers over a year ago. My uptime, bills, and stress level are all much improved.
Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)
#64The AWS status page[1] is showing problems for EC2 East as of a few minutes ago. This might be is a more widespread issue. EDIT: Various non-Heroku EC2-East-based sites (e.g. Quora) seem to be down as well, lending more evidence to this being an EC2/EBS outage. 1: http://status.aws.amazon.com/
Also Amazon Relational Database Service (N. Virginia) is unavailible.
Seems like its snowballing.
Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)
#65Earlier quoted context omitted.
It's not just ELB.
It looks like Amazon are worse at reporting their outages than Heroku...
Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)
#66Everyone should take a page from the book of Netflix right now. It's pretty embarrassing to be anyone that's entirely down and can't do a thing about it due to an EC2 outage. How do you explain to your customers/users/etc that you were down and have absolutely no control of when you will be back online? How can you explain it to yourself?
Because this shit is really hard especially when you're trying to build a product at the same time. It's not like you can just wake up one day and say "I'm gonna go build a fully fault tolerant distributed system that works across multiple data centers!" and then you're done by the time you go to sleep. Go actually talk to some Netflix engineers. They'll tell you the same thing.
Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)
#67Everyone should take a page from the book of Netflix right now. It's pretty embarrassing to be anyone that's entirely down and can't do a thing about it due to an EC2 outage. How do you explain to your customers/users/etc that you were down and have absolutely no control of when you will be back online? How can you explain it to yourself?
Because this shit is really hard especially when you're trying to build a product at the same time. It's not like you can just wake up one day and say "I'm gonna go build a fully fault tolerant distributed system that works across multiple data centers!" and then you're done by the time you go to sleep. Go actually talk to some Netflix engineers. They'll tell you the same thing.
Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)
#68Seems like someone posted a link to a project that made self-hosting a previously Heroku hosted site simple, but I can't find it now... ...would be cool if there was a Linux package (or distro) that you could boot-up and then just change your git remote to and have your app up-and-running on your own hardware.
BTW, while the point is to enable private paas, you won't get around the issues that hit sites like this without heeding all the warnings and recommendations about building in redundancy for high availability.
This was noted well in this post: http://www.newvem.com/blog/main/2012/06/aws-cloud-best-pract...
"""It is a lot cheaper to add 1% uptime to a 95% SLA than it is to add 0.09% to a 99.9% SLA. Cloud application vendors (SaaS) need to pay very close attention to the additional resources that are invested in order to support a 99.9XX…% uptime SLA, and perhaps build it into their pricing plans."""
Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)
#69Earlier quoted context omitted.
Half the startup world is offline right now, yet the AWS status page is all greens with one "info" notice about EBS (which I doubt is the issue here). I'm glad to have moved off a fully AWS stack back to my own servers over a year ago. My uptime, bills, and stress level are all much improved.
Sounds like half of the startup world hasn't spent the time or the money to design their service to be redundant on top of AWS.
Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)
#70Everyone should take a page from the book of Netflix right now. It's pretty embarrassing to be anyone that's entirely down and can't do a thing about it due to an EC2 outage. How do you explain to your customers/users/etc that you were down and have absolutely no control of when you will be back online? How can you explain it to yourself?
Because this shit is really hard especially when you're trying to build a product at the same time. It's not like you can just wake up one day and say "I'm gonna go build a fully fault tolerant distributed system that works across multiple data centers!" and then you're done by the time you go to sleep. Go actually talk to some Netflix engineers. They'll tell you the same thing.
That makes it sound like Netflix has a more reliable platform than the PaaS company.