How many times does this have to happen before heroku spreads across multiple regions?
You say this like they can just snap their fingers and provide regional services.
If not, why pay them rather than go to amazon directly?
Heroku is down again
111–120 of 134 posts
Re: Heroku is down again
#112Earlier quoted context omitted.
No matter how powerful we become as a species with our technology, we are still at the mercy of the clouds. Pretty cool if you think about it.
Or if we just built our power grid underground like rational people.
Underground cables have more expensive set up costs, lower lifetime, and higher maintenance costs. The price you pay for electricity doesn't even come close to justifying burying power lines. There's also the ecological stuff if you find that a reasonable argument. Bottom line, burying power cables just so you don't have to light a candle for a night isn't worth it.
Re: Heroku is down again
#113Can we update the title to something like "AWS US-east-1 is down" instead of just Heroku?
Appropriate title might be that "Heroku is down due to AWS outage which is down due to power failure which happened due to storms caused by moist winds colliding with hot air that was heated over the continent by sun that....". It really doesn't matter. Heroku is down. Customers don't care.
Re: Heroku is down again
#114Simple solution to this is to have a backup or failover to a non-AWS Datacenter too, basically don't be just dependent on one Datacenter. E.g. MS Azure/Google/Rackspace This not only spreads your risks but keeps your customers happy.
There's nothing simple about that. :)
Re: Heroku is down again
#115We use Heroku for our event tools, thankfully we have nothing live this weekend or this would be a disaster.
Re: Heroku is down again
#116Earlier quoted context omitted.
Even if they can't do it right away, they should communicate a plan for how they are going to tackle this recurring issue. That's the whole point behind status.heroku.com / trust.salesforce.com. They are part of a publicly traded corporation with a lot of resources. Extremely nerve wracking for new startups like ours.
I guess, it is always a good plan to have an instance deployable on Linode.
^^ This.
Re: Heroku is down again
#117Why does the AWS dashboard show all green when that is most definitely not the case? http://status.aws.amazon.com/
This was the disappointing thing for me as well. Our connectivity died around 8PM EST-ish, and I immediately went to status.aws and it said everything was normal. I then proceeded to waste half my night looking at our internal infrastructure trusting that page was accurate.
I've learned my lesson.
Re: Heroku is down again
#118Earlier quoted context omitted.
Underground cables have many problems like rats (and other vermin) or people harvesting copper/metals.
Do they? Here in Germany the entire cabling within cities is underground, only the high voltage long distance lines are above ground. I've never heard a story about people stealing underground cables (they do steal e.g. train track above ground cabling). That also wouldn't make sense, digging up those cables is much more effort than taking them down from a post. I've also never heard stories about issues with rats. P…
Rats are a common menace with all sorts of cabling. Large parts of scotland recently lots broadband due to rats eating cables (http://www.theregister.co.uk/2011/10/12/dirty_rat_downs_virg...).
Apparently it's the insulation on the wires that they like.
Re: Heroku is down again
#119Earlier quoted context omitted.
Or if we just built our power grid underground like rational people.
Underground cables have many problems like rats (and other vermin) or people harvesting copper/metals.
Really? Rats and copper harvesters?
Like it's gonna be some unprotected plastic cables 1 foot under the ground?
Re: Heroku is down again
#120Earlier quoted context omitted.
Rackspace got hit by a truck last year and went down for a while too. Not cloud != perfectly reliable.
Slightly different scenario however: the power was shut off by the fire marshall if I recollect correctly. Rackspace (and many, many other co's) tend to have functional UPS units & generators. Amazon tends to choose the cheapest datacenter facility imaginable and then these sort of failures occur. Given their size they'll inevitably fix the power issues though -- they've got the finances & they're capable to add a fe…
I found the reports about the outage - it was 2007 (so obviously much more than a year ago) but very similar to one of Amazon's recent outages - the truck took out a transformer, Rackspace fired up backup power, but cooling failed to start so Rackspace had to shut it all down to avoid melting everything.
Looks like Amazon wasn't the only one with inadequate testing of their continuity plan. And I don't think Rackspace offered alternate Availability Zones at that point.