Live data from Hacker News

Tell HN: Heroku is Down (update: recovering as of 10PM PST)

news.ycombinator.com

61–70 of 116 posts

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#61
post #53

The AWS status page[1] is showing problems for EC2 East as of a few minutes ago. This might be is a more widespread issue. EDIT: Various non-Heroku EC2-East-based sites (e.g. Quora) seem to be down as well, lending more evidence to this being an EC2/EBS outage. 1: http://status.aws.amazon.com/

Out of curiosity do we know if http://status.aws.amazon.com/ is hosted on AWS?

It's a key design constraint of status.aws.amazon.com that it not depend on any AWS services. (Or so I've heard.)

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#62
post #20

Earlier quoted context omitted.

I just got up (it's morning in Israel), and a client of mine in the US with a major, mission-critical application was screaming (rightly so) that things are down. We're already looking into alternatives -- perhaps not leaving Heroku altogether, but certainly not depending on them 100 percent. There's no way that we can entrust the business to something that can just catastrophically fail at any moment. I've been runn…

Dont overreact... heroku is using AWS which is down for many other sites right now other than heroku. This is an EBS issue that is occurring.

And my guess is his client doesn't give two excrements what the reason is. It's boolean to them; it's up or it's down.

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#63

The AWS status page[1] is showing problems for EC2 East as of a few minutes ago. This might be is a more widespread issue. EDIT: Various non-Heroku EC2-East-based sites (e.g. Quora) seem to be down as well, lending more evidence to this being an EC2/EBS outage. 1: http://status.aws.amazon.com/

Half the startup world is offline right now, yet the AWS status page is all greens with one "info" notice about EBS (which I doubt is the issue here). I'm glad to have moved off a fully AWS stack back to my own servers over a year ago. My uptime, bills, and stress level are all much improved.

Sounds like half of the startup world hasn't spent the time or the money to design their service to be redundant on top of AWS.

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#64

The AWS status page[1] is showing problems for EC2 East as of a few minutes ago. This might be is a more widespread issue. EDIT: Various non-Heroku EC2-East-based sites (e.g. Quora) seem to be down as well, lending more evidence to this being an EC2/EBS outage. 1: http://status.aws.amazon.com/

They posted an update on amazon: We continue to investigate this issue. We can confirm that there is both impact to volumes and instances in a single AZ in US-EAST-1 Region. We are also experiencing increased error rates and latencies on the EC2 APIs in the US-EAST-1 Region.

Also Amazon Relational Database Service (N. Virginia) is unavailible.

Seems like its snowballing.

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#65
post #36

Earlier quoted context omitted.

It's not just ELB.

It looks like Amazon are worse at reporting their outages than Heroku...

During the previous outage (which wasn't AWS related), Heroku's status page was down entirely (among other things, it relied on static assets from heroku.com), so I can't say I agree with that.

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#66
post #60
post #47

Everyone should take a page from the book of Netflix right now. It's pretty embarrassing to be anyone that's entirely down and can't do a thing about it due to an EC2 outage. How do you explain to your customers/users/etc that you were down and have absolutely no control of when you will be back online? How can you explain it to yourself?

Because this shit is really hard especially when you're trying to build a product at the same time. It's not like you can just wake up one day and say "I'm gonna go build a fully fault tolerant distributed system that works across multiple data centers!" and then you're done by the time you go to sleep. Go actually talk to some Netflix engineers. They'll tell you the same thing.

That's my point exactly. Everyone relies entirely (almost) on EC2 for mission critical business, and then they're left there with no outs as soon as a big outage occurs.

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#67
post #60
post #47

Everyone should take a page from the book of Netflix right now. It's pretty embarrassing to be anyone that's entirely down and can't do a thing about it due to an EC2 outage. How do you explain to your customers/users/etc that you were down and have absolutely no control of when you will be back online? How can you explain it to yourself?

Because this shit is really hard especially when you're trying to build a product at the same time. It's not like you can just wake up one day and say "I'm gonna go build a fully fault tolerant distributed system that works across multiple data centers!" and then you're done by the time you go to sleep. Go actually talk to some Netflix engineers. They'll tell you the same thing.

this. stop talking about how great a MVP is and then complain when people haven't build multi region distributed services that are fault tolerant to major platform outages.

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#68

Seems like someone posted a link to a project that made self-hosting a previously Heroku hosted site simple, but I can't find it now... ...would be cool if there was a Linux package (or distro) that you could boot-up and then just change your git remote to and have your app up-and-running on your own hardware.

This was in the recent Heroku down thread. You might be interested in Stackato (http://www.activestate.com/stackato). It is based on Cloud Foundry, with numerous enhancements, including support for Heroku buildpacks (http://docs.stackato.com/languages/buildpack.html). Heroku-in-a-box - give it a try.

BTW, while the point is to enable private paas, you won't get around the issues that hit sites like this without heeding all the warnings and recommendations about building in redundancy for high availability.

This was noted well in this post: http://www.newvem.com/blog/main/2012/06/aws-cloud-best-pract...

"""It is a lot cheaper to add 1% uptime to a 95% SLA than it is to add 0.09% to a 99.9% SLA. Cloud application vendors (SaaS) need to pay very close attention to the additional resources that are invested in order to support a 99.9XX…% uptime SLA, and perhaps build it into their pricing plans."""

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#69

Earlier quoted context omitted.

Half the startup world is offline right now, yet the AWS status page is all greens with one "info" notice about EBS (which I doubt is the issue here). I'm glad to have moved off a fully AWS stack back to my own servers over a year ago. My uptime, bills, and stress level are all much improved.

Sounds like half of the startup world hasn't spent the time or the money to design their service to be redundant on top of AWS.

Fortunately for most, they dont have customers yet

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#70
post #60
post #47

Everyone should take a page from the book of Netflix right now. It's pretty embarrassing to be anyone that's entirely down and can't do a thing about it due to an EC2 outage. How do you explain to your customers/users/etc that you were down and have absolutely no control of when you will be back online? How can you explain it to yourself?

Because this shit is really hard especially when you're trying to build a product at the same time. It's not like you can just wake up one day and say "I'm gonna go build a fully fault tolerant distributed system that works across multiple data centers!" and then you're done by the time you go to sleep. Go actually talk to some Netflix engineers. They'll tell you the same thing.

Yes, you're absolutely right. However, Netflix is distributed across multiple AZ, while Heroku has spent the last two years after their $212MM acquisition in the same AZ.

That makes it sound like Netflix has a more reliable platform than the PaaS company.

Post reply on HN