Surely I'm not the only one infuriated by their choice of blue text on a blue background?
I was all prepared to be angry, but it's actually ok, to be honest. I normally dislike dark themes.
Heroku: a follow up on last week's outage
31–40 of 48 posts
Re: Heroku: a follow up on last week's outage
#32Hey guys I got this, I speak cloudonaut. Here I'll translate it to sysadmin: An admin was doing a rolling restart that triggered a bug in the loadbalancer software. The auto restart script turned out to just make things worse by restarting it over and over (they always do), so we thought we'd just quick throw spare capacity at it, but turns out that never works in a panicked rush either. Also, our system designed to…
I know this is a joke, but from the sounds of the errors that isn't far from true. This is also basically what happened with the big two-day outage at Amazon a while back. It's always the automated processes that come back to bite you it seems. I know I have had my share of server issues, but it seems to me that many 'cloud' services out there are simply adding too many layers of abstraction that tend to make things…
Although I have my fair share of hesitance at worshiping cloud services, the fact that a service is "cloud" has nothing to do with the quality of its architecture. You can make a crucial architecture mistake designing a fleet of dedicated PHP servers talking to a MySQL cluster just as easily as you can building atop some cloud service.
Re: Heroku: a follow up on last week's outage
#33I love Heroku, but am I the only one that thinks their choice of words describing their architecture is a bit pretentious? "...streaming data API which connects the dyno manifold to the routing mesh." Give me a break!
OK, I'll bite. Instead of "dyno", they could possibly use a word like "VM". Except that they're not really virtual machines, nor are they EC2 instances. Read Only Chroot Jails plus Precompiled Application, Libraries, and Environment (ROCJPALEs?) They also have a pretty complex set of support structures that provide connectivity to databases and other resources. Perhaps someone can suggest an existing name for that, b…
Man, that's a long and contrived justification for what amounts to a pile of bullshit.
We have seen very elaborate post-mortems from google, facebook, twitter, and no least from Amazon themselves (you know, the playground that heroku builds their sandcastles in).
The aforementioned companies had no problem explaining their respective issues in plain language that every engineer did understand.
Heroku doesn't even try to explain themselves. They just throw around fantasy words without real explanations, seemingly overwhelmed by their own awesomeness (in a failure report, no less).
As an engineer I feel insulted by this pamphlet. All I can gather from it is that they screwed up and apparently somehow related to their request-routing layer. Thanks, we knew as much before reading that text.
I still have no idea what actually went wrong and how they intend to prevent it in the future. But I'll certainly advise people to avoid a company that babbles about "control rods" when their software screws up.
Re: Heroku: a follow up on last week's outage
#34The problem with Heroku is that you need to be a certain level of tech savvy to make use of their services. We're expecting them to be the A-Grade tech wizards who can give us 0 down time. They are after expecting thousands of people to trust their services and to outsource the server hosting and administration duties to them. So they tread the fine line between convenience (and related "cloud" benefits) and "I can d…
"We're expecting them to be the A-Grade tech wizards who can give us 0 down time."
Exactly. But something like this which they said makes them seem so ordinary:
"The improved status site allows users to subscribe to notifications when an incident is opened. As a result, our status site experienced unprecedented spikes of load during this incident. This high load crushed the site,"
Basically saying whatever they setup for a status site choked on sending out emails or sms, as if they were hosted on a shared server and got mentioned simultaneously on a few major sites.
Re: Heroku: a follow up on last week's outage
#35"The first root cause is related to the streaming data API which connects the dyno manifold to the routing mesh. On the dyno management side, an engineer was performing a manual garbage collection process which created an unusual record in the data stream. On the routing side, a bug in the subprocess of the router which processes the incoming stream saw the record as garbage." This is techno-babble on a scale the wor…
The only unusual terms in that paragraph are "dyno manifold" and "routing mesh", both of which are Heroku-specific technologies that Heroku users should know of. The rest is just normal systems stuff.
I can understand the words individually, but I've never before seen or heard of a manual garbage collection process creating an unusual record in a data stream. It's like a tech jargon ad lib.
Re: Heroku: a follow up on last week's outage
#36I love Heroku, but am I the only one that thinks their choice of words describing their architecture is a bit pretentious? "...streaming data API which connects the dyno manifold to the routing mesh." Give me a break!
OK, I'll bite. Instead of "dyno", they could possibly use a word like "VM". Except that they're not really virtual machines, nor are they EC2 instances. Read Only Chroot Jails plus Precompiled Application, Libraries, and Environment (ROCJPALEs?) They also have a pretty complex set of support structures that provide connectivity to databases and other resources. Perhaps someone can suggest an existing name for that, b…
If I mechanically replace the words "routing mesh" with "load balancer", I instantly know what they're talking about without losing out on any important details.
Re: Heroku: a follow up on last week's outage
#37Earlier quoted context omitted.
OK, I'll bite. Instead of "dyno", they could possibly use a word like "VM". Except that they're not really virtual machines, nor are they EC2 instances. Read Only Chroot Jails plus Precompiled Application, Libraries, and Environment (ROCJPALEs?) They also have a pretty complex set of support structures that provide connectivity to databases and other resources. Perhaps someone can suggest an existing name for that, b…
If I mechanically replace the words "routing mesh" with "load balancer", I instantly know what they're talking about without losing out on any important details.
Other than the fact that a load balancer is generally a monolithic piece of hardware. The failure modes are well defined, but most of them result in catastrophic outages.
I'm going to assume their routing mesh has many points of ingress and a larger number of exit paths (the dyno manifold), but that the nodes they've got participating in the mesh are actually in some sort of mesh topology (or form a connected graph).
This has the upside that if you lose several nodes in the mesh you probably haven't lost a path to any dynos. If you lose a whole AZ you can spin up new dynos in one of the existing ones and reconfigure the mesh quickly. My experience with loadbalancers, especially big load balancers is that updating a large swath of VIPs is NOT a fast operation (although you would start failing health checks on the missing nodes pretty quickly, adding new capacity to replace them is hard).
The mesh has the downside that the failure modes are a lot more complicated. Oh, and nobody knows what the hell you're talking about.
Of course, I could be wrong. They could just be using NetScalers (or ELB) and calling it a "routing mesh".
Re: Heroku: a follow up on last week's outage
#38Earlier quoted context omitted.
How else would you describe them?
How about words like "ec2 instances", "erlang processes", "haproxy", "nginx" and similar stuff that was likely involved in the incident? If they're too embarrassed to tell what happened then they should just keep quiet. Don't insult your customers with handwavy bullshit bingo, that just leaves a sour taste in everyones mouth. Just imagine the hilarity when the PHB asks his inhouse engineer to translate this "post-mor…
Amazon also has its own fancy vocabulary but instead of cool sounding words like manifold they prefer short acronyms.
Things like EC2 RDS AWS S3 EBS EMR IAM AMI SQS SNS SES HPC VPC
But Amazon is the reference in cloud hosting and these terms are well understood in the field.
Heroku also had to coin some words to describe their architecture. But frankly this outage report is worthy of an Hollywood hacker movie:
"A manual garbage collection process which created an unusual record in the data stream" Wow!
Re: Heroku: a follow up on last week's outage
#39Earlier quoted context omitted.
Read up on the heroku architecture. These are the terms used. The manual garbage collection wasn't the problem. An unexpected data structure created by garbage collection wasn't handled in a fault tolerant manner.
All the reading in the world will not change the fact they pick retarded names. Routing mesh? We call that a cluster of load balancers in the real world.
Well, shucks, wuts a load balancer? Sounds like some kinda doofus trying to keep a pile of shit on their head from tipping over.
Jargon is jargon, and in my part of the world, you're what we call a dumb-ass. Or is that too retarded for you? Nah, I expect there ain't a name too retarded for you.
Re: Heroku: a follow up on last week's outage
#40The problem with Heroku is that you need to be a certain level of tech savvy to make use of their services. We're expecting them to be the A-Grade tech wizards who can give us 0 down time. They are after expecting thousands of people to trust their services and to outsource the server hosting and administration duties to them. So they tread the fine line between convenience (and related "cloud" benefits) and "I can d…
I'm pretty sure they get away with it . Until we see a good cloud provider this will remain true, like the retarded pricing on current clouds.