Dear Heroku: Quit blaming all of us when you fail. Do this instead…

I apologize if anyone felt it was abusive or insulting, it certainly wasn't intended that way. I'm very passionate about holding myself directly accountable for the entirety of my user's experience with what they paid me for. It's possible that the message suggested by the OP would improve the user's experience, but I don't see how and the OP didn't make a case for that.

Instead the article read to me as if the benefit of displaying this message is that the user's frustration might be allowed to shift to the sub-contracted vendor. I find it hard not to be infuriated by that idea.

And yeah, I think it means it's the most highly rated... or at least something very close to that.

I agree with the other commenter that the analogies are getting stretched but I'll bite:

1. You haven't paid them for anything yet.

2. Wrong question!!! It's an opportunity. If I showed up for lunch and despite power being out (probably on the entire block or neighborhood) the proprietors were set up outside making cold sandwiches next to a sign that said "Sorry, power's out so only egg salad" I'd be thrilled. Here are people single-mindedly devoted to my experience.

That's really my point. It's an attitude problem. I want to spend my money with people who hustle when it hurts, and I want to do that for my users. I'm not saying it's not "fair" to close shop and blame the other guy. Sure it's fair, but the person who cares more is gonna eat (or make, in this analogy) your lunch... and the world will be a better place for it.

Failover across geographically distributed datacenters is a challenge that doesn't get talked about all that much.

As a small company you probably aren't able to easily get your own IP block allocated (that I know of) so BGP [0] isn't really an option and the best you can do is probably DNS switching. Use a good DNS provider and set your TTLs to something low like 30 seconds or 1 minute. Then when you have an outage, change the DNS entry to point to a secondary datacenter, which would have a static error page or a reduced-functionality site. There seems to be some debate around whether low DNS TTLs increase users' request times, but we haven't seen it.

There are some companies that will handle the monitoring and switchover for you (Dyn comes to mind) but we prefer to manually switchover for the time being. We have a Big Red Button sinatra app that reports the status of the site and allows you to fail over to the secondary and recover when the primary returns; I'm planning on open sourcing it once it gets some documentation.

I think the reason failover doesn't get talked about as much in the startup world is just because it's hard to do and the costs are disproportionately high for a small company unless availability is really critical to you. For most people, just using multiple availability zones on EC2 is probably sufficient.

[0] http://ajohnstone.com/achives/high-availability-across-multi...

It is their fault. The company could have used global load balancing with the app hosted in multiple geographically distributed data centers in multiple national jurisdictions through multiple independent providers. This would ensure that the earthquake which leveled Amazon's California data center and the flood which took out Hetzner's data center in Germany and the martial law declaration which took out Linode's Japanese data center and the bankruptcy which closed down Rackspace's Amsterdam data center has no operational impact on Peer1's data center in New York where the service continues uninterrupted. As the data is fully synchronized between all data centers, the company keeps running, all customers are online, and the company can work on setting up additional redundancy in a Canadian data center to make up for the others which were lost.

Yes, this costs money. It's why people accustomed to getting everything for free on the internet can't fathom why larger companies charge six or seven figures for a service that they could roll out themselves by installing an open source package on some Linode VM. If you're paying that kind of money for the reliability, it's because you're extending a promise to your end customers, and the service contract you receive from your provider should come with lots of guarantees and financial penalties if the conditions warranting the price tag aren't met.

I don't think anyone should take it upon themselves to say "Nobody cares" about the relevant details of an error. It's not just an issue of blame, but what the end customer can do with the information. The information that it is the platform that failed temporarily is useful for the end customers because they don't have to lose too much confidence in the software vendor because their code is failing. It lets the customers know the problem is probably temporary. Customers do care because a lot of them can interpret that level of error detail, and it is actionable, useful information to them. Heroku should at least give developers the option of finer grained error messages.

>In the end, you chose to host your site on a platform that went down.

That's only partially true, there are limited viable options for platform hosts out there, and virtually no one has cracked the 100% up-time challenge, so there isn't any degree to which a software developer could choose a host with 100% up-time if it doesn't exist. Thus they're not at fault for choosing the "wrong" host or not creating something that is astronomically difficult themselves.