Heroku: a follow up on last week's outage
status.heroku.com
Heroku: a follow up on last week's outage
1–10 of 48 posts
Re: Heroku: a follow up on last week's outage
#2I tend to feel much less angry about outages and such when the businesses take time to explained what happened, of course this doesn't justify the lack of up-time but I like the gesture!
Re: Heroku: a follow up on last week's outage
#3I love Heroku, but am I the only one that thinks their choice of words describing their architecture is a bit pretentious?
"...streaming data API which connects the dyno manifold to the routing mesh."
Give me a break!
Re: Heroku: a follow up on last week's outage
#4I love Heroku, but am I the only one that thinks their choice of words describing their architecture is a bit pretentious? "...streaming data API which connects the dyno manifold to the routing mesh." Give me a break!
How else would you describe them?
Re: Heroku: a follow up on last week's outage
#5I love Heroku, but am I the only one that thinks their choice of words describing their architecture is a bit pretentious? "...streaming data API which connects the dyno manifold to the routing mesh." Give me a break!
[deleted]
Re: Heroku: a follow up on last week's outage
#6Quite a few Erlang gotchas in those notes. Fault tolerant systems are really hard to design even when you know what you're doing and are using the best language for it (Erlang). Erlang aside, it seems the higher level architecture may need a rethink if one bad record can bring down the whole thing.
Re: Heroku: a follow up on last week's outage
#7I love Heroku, but am I the only one that thinks their choice of words describing their architecture is a bit pretentious? "...streaming data API which connects the dyno manifold to the routing mesh." Give me a break!
This is classical geek owning up. My first thought was, this is written with two purposes:
1) to prevent the average, non-technical person from understanding it ("Phew, I'm glad these guys are figuring out this stuff and not me - that's why I host with them. I don't even know what a 'dyno manifold' is!")
2) to show management how smart we are and that you still need us ("because who else is going to figure this 'routing mesh' stuff out if you fire those responsible for the outage")
A simple "we're sorry and we've given 10 lashes to the engineer performing the manual garbage collection" may have been a better approach.
Having said that, I still think Heroku is awesome.
Re: Heroku: a follow up on last week's outage
#8I love Heroku, but am I the only one that thinks their choice of words describing their architecture is a bit pretentious? "...streaming data API which connects the dyno manifold to the routing mesh." Give me a break!
This is classical geek owning up. My first thought was, this is written with two purposes: 1) to prevent the average, non-technical person from understanding it ("Phew, I'm glad these guys are figuring out this stuff and not me - that's why I host with them. I don't even know what a 'dyno manifold' is!") 2) to show management how smart we are and that you still need us ("because who else is going to figure this 'rout…
Read up on the heroku architecture. These are the terms used.
The manual garbage collection wasn't the problem. An unexpected data structure created by garbage collection wasn't handled in a fault tolerant manner.
Re: Heroku: a follow up on last week's outage
#9I love Heroku, but am I the only one that thinks their choice of words describing their architecture is a bit pretentious? "...streaming data API which connects the dyno manifold to the routing mesh." Give me a break!
How else would you describe them?
"connects the tachyon emitter to the warp nacelles"?
Re: Heroku: a follow up on last week's outage
#10"The first root cause is related to the streaming data API which connects the dyno manifold to the routing mesh. On the dyno management side, an engineer was performing a manual garbage collection process which created an unusual record in the data stream. On the routing side, a bug in the subprocess of the router which processes the incoming stream saw the record as garbage."
This is techno-babble on a scale the world has never seen!