Heroku Dynos Unable to Start

I know it's popular to be a "developer-centric" organization these days, but please, please, don't schedule risky maintenance operations for 10 AM on a Monday, when your tools are in heavy usage.

Every SaaS product I've built has analyzed traffic, and performed migrations/deployments at off-hours (generally, after midnight in our dominant time zone). In this case, the outage may have resulted in hundreds (thousands?) of paged administrators across the world, but at least fewer end-users would have been affected.

Also, at scale, it's a good idea to deploy to a single cluster/zone first, and check error rates before deploying to the larger environment.

It's pretty scary that the 'professionals' to whom I've trusted my business aren't more savvy when it comes to high availability...

Re: Heroku Dynos Unable to Start

#6

Post by stefangomez » Mon, Jun 23, 2014, 6:33 PM UTC

shravan wrote: post #2

Request queueing on New Relic is going haywire on our app right now.

PX dynos seem to have not been affected

Re: Heroku Dynos Unable to Start

#7

Post by kawsper » Mon, Jun 23, 2014, 6:36 PM UTC

sparkman55 wrote: post #5

I know it's popular to be a "developer-centric" organization these days, but please, please, don't schedule risky maintenance operations for 10 AM on a Monday, when your tools are in heavy usage. Every SaaS product I've built has analyzed traffic, and performed migrations/deployments at off-hours (generally, after midnight in our dominant time zone). In this case, the outage may have resulted in hundreds (thousands?)…

Are you sure that Heroku is peaking 10 AM on a Monday?

I believe they have off-hours, but as an international product it could be interesting to know when. I know a lot of European products with European users that all depend on Heroku.

Re: Heroku Dynos Unable to Start

#8

Post by sparkman55 » Mon, Jun 23, 2014, 6:46 PM UTC

kawsper wrote: post #7

sparkman55 wrote: post #5

I know it's popular to be a "developer-centric" organization these days, but please, please, don't schedule risky maintenance operations for 10 AM on a Monday, when your tools are in heavy usage. Every SaaS product I've built has analyzed traffic, and performed migrations/deployments at off-hours (generally, after midnight in our dominant time zone). In this case, the outage may have resulted in hundreds (thousands?)…

Are you sure that Heroku is peaking 10 AM on a Monday? I believe they have off-hours, but as an international product it could be interesting to know when. I know a lot of European products with European users that all depend on Heroku.

I'm not sure that Heroku is peaking at 10 AM on Monday, but I'm reasonably sure it's not a traffic nadir for them.

For example, 6 PM on Saturday PDT is 1 AM on Sunday GMT and 9 AM on Saturday in Hong Kong. This would almost certainly be a time of lower deployments / requests globally.

Re: Heroku Dynos Unable to Start

#9

Post by sprite » Mon, Jun 23, 2014, 7:04 PM UTC

Started getting tons of emails from end users. Went to investigate and was greeted with this:

! ! Heroku has temporarily disabled this feature, please try again shortly. ! See http://status.heroku.com for current Heroku platform status.

Hopefully they will be backup up soon. Also wonder if we will get any sort of reimbursement? I currently spend around $3k/month with Heroku.

Re: Heroku Dynos Unable to Start

#10

Post by sprite » Mon, Jun 23, 2014, 7:06 PM UTC

shravan wrote: post #2

Request queueing on New Relic is going haywire on our app right now.

I can't even get in to New Relic.