Heroku: a follow up on last week's outage

Hey guys I got this, I speak cloudonaut. Here I'll translate it to sysadmin:

An admin was doing a rolling restart that triggered a bug in the loadbalancer software. The auto restart script turned out to just make things worse by restarting it over and over (they always do), so we thought we'd just quick throw spare capacity at it, but turns out that never works in a panicked rush either. Also, our system designed to handle outage notifications wasn't capacity planned, like, at all.

Pretentious is an understatement.

This seems to be an unfortunate attempt to apply Corporate Speak to a technical announcement; "Let's see how many paragraphs we can fill with technical-sounding gibberish without actually telling anything..."

All the reading in the world will not change the fact they pick retarded names.

Routing mesh? We call that a cluster of load balancers in the real world.

OK, I'll bite.

Instead of "dyno", they could possibly use a word like "VM". Except that they're not really virtual machines, nor are they EC2 instances. Read Only Chroot Jails plus Precompiled Application, Libraries, and Environment (ROCJPALEs?) They also have a pretty complex set of support structures that provide connectivity to databases and other resources. Perhaps someone can suggest an existing name for that, but I know of none.

Instead of "manifold", perhaps they could use the word "cluster". Except it's not really a cluster, it's a set of distributed clusters. And nodes in a cluster are typically machines. The nodes in the dyno manifold aren't machines, virtual machines, they're ROCJPALEs. You could use the word "array", but again, it's not really an array. It's a multi-layered, geographically distributed structure of co-hosted application jails. "Manifold" seems as good a term as any.

"Streaming" seems like a good word. It's specifically relevant to this incident... they describe how the API is not atomic; that each message is built on top of the previous entries, and the data structures are implicit in the stream. That sounds like the definition of "streaming" to me.

"API" seems like a widely accepted term. They could've described it as a "protocol", perhaps. But neither seems more jargony than the other.

"Data"... well I suppose "streaming API" without the data would work. But it serves to differentiate it from a streaming video protocol.

"Mesh" has a very specific meaning. It means that you have a set of nodes that are connected peer-to-peer and that messages travel through the network by hopping from node to node. I'm assuming that their routing layer is organized in this way.

"Routing" is also pretty well defined. Requests come in and need to be sent to the machine that can serve responses to it. What would you call that instead of routing?

I feel like people who object to this kind of language are the same folks who object to the word "cloud". People don't take the time to understand different strategies to provisioning and application hosting APIs, and then think these words don't mean anything. Yeah, salespeople use the word to hustle the Same Old Shit, but it also actually means something to people like us who are actually building stuff.

I know this is a joke, but from the sounds of the errors that isn't far from true. This is also basically what happened with the big two-day outage at Amazon a while back. It's always the automated processes that come back to bite you it seems.

I know I have had my share of server issues, but it seems to me that many 'cloud' services out there are simply adding too many layers of abstraction that tend to make things very, very touchy to any small issue occurring. Because of this I try to keep my server stacks/frameworks as basic as possible while still implementing performance oriented services like NoSQL, caching, etc.