131–140 of 311 posts
Per the linked dashboard, some instances in a single AZ in a single Region are having storage issues. Calling EC2 "down" is a bit dramatic, provided AMZN are being sincere with their status reports. Any system that can competently fail over to another AZ will be unaffected.
Earlier quoted context omitted.
shouldn't amazon themselves have architected their own app to be able to move around? I get tired of the snipes from people that "well, you're doing it wrong", as if this is trivial stuff. But if Amazon themselves aren't even making their AWS console redundant between locations, how easy/straightforward is it for anyone else? To what extent is this just "the cobbler's kids have no shoes?"
If it's systematically difficult to do it correctly, then the system is wrong.
Our app is down because it's hosted on Heroku and it's frustrating because it seems like N Virginia is the least reliable Amazon datacenter. Every year it seems to go down a couple times for at least a couple hours. Heroku should offer a choice between N Virginia and Oregon hosting (I think they're almost comparable in price nowadays). That way people who want more uptime/reliability can choose Oregon. Sure it will b…
You have burstable network connections which by their nature, will have hard limits (you can't burst above 10Gbps on a 10Gbps pipe, for example; even assuming the host machine is connected to a 10Gbps port).
Burstable (meaning quite frankly, over-provisioned) disk and CPU resources.
And if any piece fails, you may well have downtime...
It is always surprising to me, that people feel that layering complexity upon complexity, will result in greater reliability.
pg, or someone else from HN... Could you please edit this title for accuracy? Maybe, "Poorly designed sites taken out because of problems in one Amazon availability zone."
That's editorializing. For many sites, a single server in a single zone (e.g., a non redundant server, an instance, a slice, a VM, whatever) is the right decision for ROI. For many sites, the money spent on redundancy could be better spent on, say, Google Adwords, until they're big enough that a couple hours downtime has irreplaceable costs higher than the added costs of redundancy (dev, hosting, admin) for a year.
As https://twitter.com/DEVOPS_BORAT says, At conference you can able tell cloud devops by they are always leave dinner for respond to pager. Also, What is happen in cloud is stay in cloud because nobody can able reproduce outside of cloud. (And many other relevant quotes.)
To be fair, AWS downtime always make the news because they affect a lot of majors websites, but that doesn't mean an average sysadmin (or devops, whatever) would do better in term of uptime with his own bay and his toys.
Just one multi-AZ RDS instance claimed the automatic failover. However, the 200+ alerts over the automatic failover due to internal DNS changes to point to the new master shows that things aren't as easily described by the RDS DB Events log.
Some instances reported high disk I/O (EBS backed) via the New Relic agent (the console still has some issues).
So far, this is what I see from my side.
This would be a great time to post a guide to architecting systems for failover using AWS. Anyone got a great guide?
We (Netflix) have done a bunch of presentations on it which are on our slideshare page and across the internet. After this issue is over I can give a longer answer. In short, we've just evacuated the affected zone and are mostly recovered.
And +1 for the slideshare page.
Their techblog is also worth following: http://techblog.netflix.com/