Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Heroku is Down (update: recovering as of 10PM PST)

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 116 posts

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#71

Post by Cushman »

zeeg wrote: post #47

Everyone should take a page from the book of Netflix right now. It's pretty embarrassing to be anyone that's entirely down and can't do a thing about it due to an EC2 outage. How do you explain to your customers/users/etc that you were down and have absolutely no control of when you will be back online? How can you explain it to yourself?

"We're sorry about the current downtime. We know some of you are frustrated, so we thought we'd take a moment to explain why this happened.

"Running a web server is very expensive. After we've built the site, if we want to keep it running, someone needs to be on-call 24 hours a day. That means at least one full-time staff member who does nothing else-- more if we want them to stay sane.

"To save us and you some money, we've contracted maintenance of our servers out to a third-party service. This is great for us, since they run it more reliably than we could, and it's great for you, because it costs a lot less. But the downside is that things still break sometimes, and when they do, it's completely out of our hands. We're left waiting for things to get fixed just like you are.

"So we understand your frustration; we're frustrated too. But unfortunately, downtimes do happen. Guaranteeing our service 100% of the time would cost hundreds of thousands of extra dollars per year, and for most of our users, that's simply not worth the cost. Our provider guarantees 99.[nines]% reliability for much less money, and this is the 0.01%.

"If you have something that absolutely must get done, shoot us an email right now and we'll take care of it for you as soon as the site is active again.

"Although this is technically out of our hands, we aren't trying to shift the blame; we made this decision with open eyes, and we stand by that decision. Again, we sincerely apologize for the temporary inconvenience. We hope we can make it up to you with some new features we'll be rolling out this month :)"

It doesn't have to be that hard.

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#72

Post by biot »

atlasom wrote: post #48

Can we get this title renamed to AWS outage as the problem is not Heroku.

It may be an AWS outage as the cause, but ultimately it's Heroku's problem. They're the ones touting:

  "Erosion-resistant architecture.

   Heroku takes full responsibility for your app's health,
   keeping it up and running through thick and thin..."
The thick happened today, something eroded, and people are holding them to their word: full responsibility for their app's health. Heroku could do load balancing between multiple independent providers rather than be solely dependent on [one region of?] AWS.

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#73

Post by ctrand »

ajasmin wrote: post #53
michaelfairley wrote: post #2

The AWS status page[1] is showing problems for EC2 East as of a few minutes ago. This might be is a more widespread issue. EDIT: Various non-Heroku EC2-East-based sites (e.g. Quora) seem to be down as well, lending more evidence to this being an EC2/EBS outage. 1: http://status.aws.amazon.com/

Out of curiosity do we know if http://status.aws.amazon.com/ is hosted on AWS?

I thought I read somewhere that it was on Rackspace

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#74

Post by xxpor »

hardoncollider wrote: post #42

That's odd, because I thought the error occurred in my application. No? You sure? Just look at this error message, "An error occurred in the application and your page could not be served." Oh, so you mean the error message aren't accurate? That even though the error is actually above Heroku, a much greater failure, that my application will still be blamed, and that's what my users and bosses are going to see? And you…

Isn't it your fault you didn't build a fault tolerant application? The first rule of building services is assume everything is broken.

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#75

Post by mnutt »

zeeg wrote: post #47

Everyone should take a page from the book of Netflix right now. It's pretty embarrassing to be anyone that's entirely down and can't do a thing about it due to an EC2 outage. How do you explain to your customers/users/etc that you were down and have absolutely no control of when you will be back online? How can you explain it to yourself?

Maybe you tell them that they won't be able to trade cat pictures for some time. And that the outage happened because you didn't overinvest in infrastructure so that you could keep prices down and roll out some new features faster.

Even if you aren't a cat picture site, many startups can deal with some downtime occasionally and it's the right tradeoff to make.

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#77

Post by ctrand »

bdesimone wrote: post #56

Dear Heroku -- I know it's my job to make sure my site is available (/thread). However, I think I speak for most enterprise customers when I say I will throw money at your company the second you come up with a multi-zone/highly-available.

I spoke to one of the guys from Heroku recently and he said they are working on it but he couldn't give me a date.

It couldn't come soon enough for us, but also for Heroku as AppFog seem to have its foundations built on a multi-zone/region/provider architecture.

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#78

Post by fusiongyro »

res0nat0r wrote: post #63
dangrossman wrote: post #34

Earlier quoted context omitted.

Half the startup world is offline right now, yet the AWS status page is all greens with one "info" notice about EBS (which I doubt is the issue here). I'm glad to have moved off a fully AWS stack back to my own servers over a year ago. My uptime, bills, and stress level are all much improved.

Sounds like half of the startup world hasn't spent the time or the money to design their service to be redundant on top of AWS.

That's hardly surprising. The groupthink on cloud services is that they magically make everything scalable, cheap and easy. Making redundant software is hard work.

Re: Tell HN: Heroku is Down (update: recovering as of 10PM PST)

#80

Post by jaggederest »

andrewcooke wrote: post #46
andrewvc wrote: post #28

Earlier quoted context omitted.

That's 99.97 for the last month . Which is ridiculous, 9s for most services use years as standard. Of course if heroku did that they wouldn't look so good.

if you have a year of 99.97 months, don't you have a 99.97 year? how does having a shorter time make the numbers look better? [edit: i realise they could be hiding worse months, but i don't think that's what the post i am replying to meant. perhaps i am reading it wrong.]

Parent post was saying that, without taking into account the entire year, you can have one month that is terrible and the rest quite good.

The measurement in question is supposed to be about consistency.

Post reply on HN