Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku: a follow up on last week's outage

status.heroku.com

Post reply on HN
Print view View original

41–48 of 48 posts

Re: Heroku: a follow up on last week's outage

#41

Post by joeyh »

cabinguy wrote: post #26
Negitivefrags wrote: post #10

"The first root cause is related to the streaming data API which connects the dyno manifold to the routing mesh. On the dyno management side, an engineer was performing a manual garbage collection process which created an unusual record in the data stream. On the routing side, a bug in the subprocess of the router which processes the incoming stream saw the record as garbage." This is techno-babble on a scale the wor…

I like the part where some guy/gal with a stop watch was keeping records of every minute.

Their interactive engineer mesh stream handles this time keeping.

(That is, probably an irc channel..)

Re: Heroku: a follow up on last week's outage

#42

Post by pja »

kennystone wrote: post #6

Quite a few Erlang gotchas in those notes. Fault tolerant systems are really hard to design even when you know what you're doing and are using the best language for it (Erlang). Erlang aside, it seems the higher level architecture may need a rethink if one bad record can bring down the whole thing.

It looks like the error recovery code wasn't well tested. Error recovery code in distributed systems is some of the hardest code to test effectively mind.

The thundering herd of recovery is especially difficult to cope with: your error recovery code can work just fine for normal outages but then fail completely when faced with just a few more components going dark.

Re: Heroku: a follow up on last week's outage

#43

Post by pja »

andrewcooke wrote: post #12

weird. they had a problem caused by a series of bugs, yet the word "test" doesn't appear anywhere in that page.

Testing distributed systems is much, much harder than doing so on a monolithic codebase. The number of failure modes goes up very rapidly with the number of nodes in the system & your code has to (in principle) cope with every possible one.

Re: Heroku: a follow up on last week's outage

#44

Post by tedunangst »

jsolson wrote: post #37
tedunangst wrote: post #36

Earlier quoted context omitted.

If I mechanically replace the words "routing mesh" with "load balancer", I instantly know what they're talking about without losing out on any important details.

Other than the fact that a load balancer is generally a monolithic piece of hardware. The failure modes are well defined, but most of them result in catastrophic outages. I'm going to assume their routing mesh has many points of ingress and a larger number of exit paths (the dyno manifold), but that the nodes they've got participating in the mesh are actually in some sort of mesh topology (or form a connected graph).…

Oh, and nobody knows what the hell you're talking about.

Yes. It's a level of detail that borders on obfuscation.

fwiw, I've always used the term "load balancer" to also refer to two redundant load balancing machines. (If I worked with more complex load balancers, I doubt I'd stop.) In the general sense, it just means "the apparatus that balances the load".

Re: Heroku: a follow up on last week's outage

#45

Post by andrewcooke »

pja wrote: post #43
andrewcooke wrote: post #12

weird. they had a problem caused by a series of bugs, yet the word "test" doesn't appear anywhere in that page.

Testing distributed systems is much, much harder than doing so on a monolithic codebase. The number of failure modes goes up very rapidly with the number of nodes in the system & your code has to (in principle) cope with every possible one.

true, but it sounds like they (and perhaps you) have never even heard of the chaos monkey.

Re: Heroku: a follow up on last week's outage

#46

Post by pja »

andrewcooke wrote: post #45
pja wrote: post #43

Earlier quoted context omitted.

Testing distributed systems is much, much harder than doing so on a monolithic codebase. The number of failure modes goes up very rapidly with the number of nodes in the system & your code has to (in principle) cope with every possible one.

true, but it sounds like they (and perhaps you) have never even heard of the chaos monkey.

Randomly killing instances wouldn't have detected this particular failure mode as far as I can see, since the error lay in the inability to resurrect a failed process under certain circumstances.

Re: Heroku: a follow up on last week's outage

#47

Post by FireBeyond »

ajasmin wrote: post #38
moe wrote: post #24

Earlier quoted context omitted.

How about words like "ec2 instances", "erlang processes", "haproxy", "nginx" and similar stuff that was likely involved in the incident? If they're too embarrassed to tell what happened then they should just keep quiet. Don't insult your customers with handwavy bullshit bingo, that just leaves a sour taste in everyones mouth. Just imagine the hilarity when the PHB asks his inhouse engineer to translate this "post-mor…

Amazon also has its own fancy vocabulary but instead of cool sounding words like manifold they prefer short acronyms. Things like EC2 RDS AWS S3 EBS EMR IAM AMI SQS SNS SES HPC VPC But Amazon is the reference in cloud hosting and these terms are well understood in the field. Heroku also had to coin some words to describe their architecture. But frankly this outage report is worthy of an Hollywood hacker movie: "A man…

Heroku had to coin some words of their own to "mask" the fact that their services are but engineering on top of the AWS stack (which isn't to belittle the effort involved).

Re: Heroku: a follow up on last week's outage

#48

Post by erikpukinskis »

moe wrote: post #33
erikpukinskis wrote: post #27

Earlier quoted context omitted.

OK, I'll bite. Instead of "dyno", they could possibly use a word like "VM". Except that they're not really virtual machines, nor are they EC2 instances. Read Only Chroot Jails plus Precompiled Application, Libraries, and Environment (ROCJPALEs?) They also have a pretty complex set of support structures that provide connectivity to databases and other resources. Perhaps someone can suggest an existing name for that, b…

Man, that's a long and contrived justification for what amounts to a pile of bullshit. We have seen very elaborate post-mortems from google, facebook, twitter, and no least from Amazon themselves (you know, the playground that heroku builds their sandcastles in). The aforementioned companies had no problem explaining their respective issues in plain language that every engineer did understand. Heroku doesn't even try…

Are you a Heroku customer? I am, and I understand everything they said, and I appreciate that they went into detail about what happened.
Post reply on HN