Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku is down

heroku.com

Post reply on HN
Print view View original

81–90 of 107 posts

Re: Heroku is down

#81

Post by cardmagic »

kitsune_ wrote: post #61

We've recently begun to move our entire infrastructure to external providers (vps and cloud) because we felt that we couldn't guarantee a sufficiently high level of quality and had to devote too much time and money to operations. In over four months we've probably already had more downtime than in the past 5 years. Despite paying quite a lot for "redundant", clustered offerings. There was one nasty bug in the hypervi…

http://AppFog.com has ruby, python and node, multiple regions today: Ireland, Singapore and US, adding Rackspace and HP soon. All free and backed by CloudFoundry

Re: Heroku is down

#82

Post by joevandyk »

jsprinkles wrote: post #77
joevandyk wrote: post #76

Earlier quoted context omitted.

One of the problems with using multiple providers is you can't use any of the specific features of a provider. For example, I really like AWS's security groups and ELBs. Those serve as my firewall and my load balancer and SSL terminator. Replicating the application to another service means configuring and testing all that on my own. If I use heroku and use their logging system, then replicating it to another provider…

That's exactly why you don't want to use those features, for what it's worth.

That means that I need to know about (and program chef to do):

logging

security hardening

firewalls

backups (both point in time and complete backups, both for database and for any other artifacts like image uploads)

testing backup recovery (both the point in time and complete backups)

replication

HA

LB

SSL termination

postgresql configuration

monitoring (security, system stats, application availability, individual process running)

performance analysis (both at application and system level)

method for deploying updates (to the application and to the system)

and the list goes on... you could easily make a career out of focusing on postgresql configuration, for example. these aren't simple things to do.

Different providers have different limitations on what you can do here. For example, AWS doesn't support multicast, which limits your options for doing High Availability.

Re: Heroku is down

#83

Post by chao- »

jsprinkles wrote: post #72
patio11 wrote: post #63

Earlier quoted context omitted.

Ask a simple question, get a simple answer: No. This is no knock on Heroku, either. No hosting solution can make that easy: it is an enterprise requirement which implies six figures of investment and a dedicated ops team with no newbies on it.

I'm sorry, but that's completely untrue. I can't speak for using Heroku specifically, but in the end hosting is just running an app. It's fairly trivial to make an app multi-provider these days through a plethora of methods. DNS and a low TTL is on the easiest-to-approach end if your app is designed for it and aware of the complications, which implies thinking about your database and other supporting architecture fro…

I am intrigued. Do you have any quick links on hand, or pointers on how one would go about learning how to set up something like this? Or just learning about how to distribute services in general?

That is to say, I know my way around Linux, have set up single/standalone servers for all sorts of services, but have never known remotely where to begin on the distributed side of things, much less understand what is required to makes something "designed to be multihomed".

Re: Heroku is down

#84

Post by joevandyk »

ralphleon wrote: post #62

It's so scary to think that our entire startup almost died with this glitch. Our enterprise customers use our product in the mornings... this happened at 9-fucking-am right during our peak hours. Obviously it's our fault for not having a redundant system / for trusting heroku. Though I still can't help but be a little pissed as I email 100 people about how sorry I am for their service disruption. Heroku didn't email…

You can subscribe to notifications on http://status.heroku.com.

Re: Heroku is down

#85

Post by grandalf »

ricardobeat wrote: post #49
grandalf wrote: post #41

Heroku is a great platform and has come a really long way -- but it could really use some competition. Some of the main issues that would be resolved quickly with competition are: - it would be possible to have the non-DB part of your heroku app be spread across multiple availability zones. - worker pricing would be much lower or based on actual CPU cycles used. - add-on providers would be vetted more thoroughly befo…

you mean like AWS, EngineYard, appfog, dotCloud?

I haven't looked at DotCloud, but the others are not really competitors.

Re: Heroku is down

#86

Post by pardner »

bitsweet wrote: post #30

What would really help in these situations is a failover to our maintenance pages or some other static page we could provide instead of showing the Heroku "Application Error". This is probably harder then it seems, especially when the outage is related to their routing infrastructure.

I agree. There is a very simple way to do this which I just posted (since it apparently hasn't occurred to Heroku). All they need to do is change the default app error message once they have confirmed they have a platform issue. So within a few minutes of an outage, a better message is displayed to our customers that does NOT make it look like we screwed up. http://blog.pardner.com/2012/06/dear-heroku-dammit-quit-blam...

Re: Heroku is down

#87

Post by [deleted] »

ralphleon wrote: post #62

It's so scary to think that our entire startup almost died with this glitch. Our enterprise customers use our product in the mornings... this happened at 9-fucking-am right during our peak hours. Obviously it's our fault for not having a redundant system / for trusting heroku. Though I still can't help but be a little pissed as I email 100 people about how sorry I am for their service disruption. Heroku didn't email…

[deleted]

Re: Heroku is down

#88

Post by jsprinkles »

joevandyk wrote: post #82
jsprinkles wrote: post #77

Earlier quoted context omitted.

That's exactly why you don't want to use those features, for what it's worth.

That means that I need to know about (and program chef to do): logging security hardening firewalls backups (both point in time and complete backups, both for database and for any other artifacts like image uploads) testing backup recovery (both the point in time and complete backups) replication HA LB SSL termination postgresql configuration monitoring (security, system stats, application availability, individual pr…

I think before you put an app on the public Internet for people to consume, you should be comfortable with everything you listed plus more. You might not be doing it for the app at hand, but you should know it at least. Especially if you're doing a solo founder thing, you might be having to implement all of these things before you can afford ops. If you're deploying services without some of the knowledge in your list, too, you're setting yourself up for trouble.

It's not an insurmountable barrier, for sure. Knowing how to operate your own code is a great feeling.

Re: Heroku is down

#89

Post by jsprinkles »

neilmiddleton wrote: post #70
kitsune_ wrote: post #61

We've recently begun to move our entire infrastructure to external providers (vps and cloud) because we felt that we couldn't guarantee a sufficiently high level of quality and had to devote too much time and money to operations. In over four months we've probably already had more downtime than in the past 5 years. Despite paying quite a lot for "redundant", clustered offerings. There was one nasty bug in the hypervi…

Heroku status is currently showing production uptime as 99.97%. That's not too shabby, and definitely not as bad as you make out.

He implied that Heroku is not his sole provider. I'm skeptical of that figure, too, as it's probably something on the Heroku sales site (99.7% to the 30-byte health check!), or it's manually updated, in which case it's probably rounded.

Re: Heroku is down

#90

Post by dholowiski »

bitsweet wrote: post #30

What would really help in these situations is a failover to our maintenance pages or some other static page we could provide instead of showing the Heroku "Application Error". This is probably harder then it seems, especially when the outage is related to their routing infrastructure.

How would that help?
Post reply on HN