Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku Dynos Unable to Start

status.heroku.com

Post reply on HN
Print view View original

11–20 of 24 posts

Re: Heroku Dynos Unable to Start

#11

Post by pardner »

Compounding the issue of irresponsibly scheduling maintenance for US zones mid-day in the US, they ALSO broke ability to scale worker processes to zero, so there was literally NO way to wind your app down and prevent worker jobs from firing off.

When the platform started getting wonky we shut off our worker dynos so the system would not fire off emails while the system is in a known-screwed-up state.

The console said workers were set to zero.

But... in our logs we watched the (supposedly off) workers continue to fire off.

Nice FUBAR all around, Heroku.

Re: Heroku Dynos Unable to Start

#12

Post by pardner »

sparkman55 wrote: post #5

I know it's popular to be a "developer-centric" organization these days, but please, please, don't schedule risky maintenance operations for 10 AM on a Monday, when your tools are in heavy usage. Every SaaS product I've built has analyzed traffic, and performed migrations/deployments at off-hours (generally, after midnight in our dominant time zone). In this case, the outage may have resulted in hundreds (thousands?)…

There is no reason they cannot perform maintenance for US zones and EU zones at different times and avoid M-F mid-day for everyone. So it's probably "just easier" for Heroku this way.

Re: Heroku Dynos Unable to Start

#15

Post by zzen »

kawsper wrote: post #7
sparkman55 wrote: post #5

I know it's popular to be a "developer-centric" organization these days, but please, please, don't schedule risky maintenance operations for 10 AM on a Monday, when your tools are in heavy usage. Every SaaS product I've built has analyzed traffic, and performed migrations/deployments at off-hours (generally, after midnight in our dominant time zone). In this case, the outage may have resulted in hundreds (thousands?)…

Are you sure that Heroku is peaking 10 AM on a Monday? I believe they have off-hours, but as an international product it could be interesting to know when. I know a lot of European products with European users that all depend on Heroku.

They have a separate EU hosting region that has separate maintenance (and uptime that's actually significantly better then US region): https://status.heroku.com/uptime?region=EU

So yeah, it was a lame time to schedule maintenance.

Re: Heroku Dynos Unable to Start

#16

Post by alrs »

zzen wrote: post #15
kawsper wrote: post #7

Earlier quoted context omitted.

Are you sure that Heroku is peaking 10 AM on a Monday? I believe they have off-hours, but as an international product it could be interesting to know when. I know a lot of European products with European users that all depend on Heroku.

They have a separate EU hosting region that has separate maintenance (and uptime that's actually significantly better then US region): https://status.heroku.com/uptime?region=EU So yeah, it was a lame time to schedule maintenance.

I disagree, strongly.

If you've architected for IAAS you have sufficient redundancy and a plan for graceful degradation. I'd rather be dealing with an outage after lunch than at 4am.

Re: Heroku Dynos Unable to Start

#17

Post by alrs »

silasb wrote: post #13

Makes zero sense to do an 2 hour update on a Monday at 10 AM PDT. Horrible horrible timing. I'll likely be getting blamed for this since I recommended Heroku.

Do you want to start working night shifts? The people who know how do this stuff command six-figure salaries and are highly in demand.

Every startup is looking for the design/dev/ops unicorn. You want to start trying to find nocturnal unicorns?

Re: Heroku Dynos Unable to Start

#18

Post by sparkman55 »

alrs wrote: post #16
zzen wrote: post #15

Earlier quoted context omitted.

They have a separate EU hosting region that has separate maintenance (and uptime that's actually significantly better then US region): https://status.heroku.com/uptime?region=EU So yeah, it was a lame time to schedule maintenance.

I disagree, strongly. If you've architected for IAAS you have sufficient redundancy and a plan for graceful degradation. I'd rather be dealing with an outage after lunch than at 4am.

This is exactly the "developer-centric" mindset that is, frankly, misguided.

Sure, I would rather deal with an outage during business hours, but that means that many of my Customers are also dealing with an outage. If the outage were at 4 AM, most of my Customers would be asleep, and wouldn't even notice the outage.

Re: Heroku Dynos Unable to Start

#19

Post by alrs »

sparkman55 wrote: post #18
alrs wrote: post #16

Earlier quoted context omitted.

I disagree, strongly. If you've architected for IAAS you have sufficient redundancy and a plan for graceful degradation. I'd rather be dealing with an outage after lunch than at 4am.

This is exactly the "developer-centric" mindset that is, frankly, misguided. Sure, I would rather deal with an outage during business hours, but that means that many of my Customers are also dealing with an outage. If the outage were at 4 AM, most of my Customers would be asleep, and wouldn't even notice the outage.

If you want NASA reliability, build a NASA-grade 24/7 operation across three timezones.

If your plan is just to have the nerds skip sleep periodically, you're an asshole.

Re: Heroku Dynos Unable to Start

#20

Post by stevepike »

alrs wrote: post #17
silasb wrote: post #13

Makes zero sense to do an 2 hour update on a Monday at 10 AM PDT. Horrible horrible timing. I'll likely be getting blamed for this since I recommended Heroku.

Do you want to start working night shifts? The people who know how do this stuff command six-figure salaries and are highly in demand. Every startup is looking for the design/dev/ops unicorn. You want to start trying to find nocturnal unicorns?

These don't need to be night shifts. Back when I worked in bigco, we had very competent ops teams around the world - we'd even send developers abroad for 3-12 month periods to train with them and share information.

Smaller companies use Heroku so we don't have to build the same expertise in house. They're able to charge a premium not because their developers have to work crazy hours, but because there's a certain base cost to having these kinds of distributed teams.

Post reply on HN