It's interesting how AWS outage didn't take down Amazon.com.
Actually temporarily it did. At least for me for about 5 min amazon.com was unreachable yesterday when i first read about this. was around 10pm EST i believe.
Amazon cloud outage takes down Netflix, Instagram, Pinterest, & more
31–37 of 37 posts
Re: Amazon cloud outage takes down Netflix, Instagram, Pinterest, & more
#32It's funny, the more I think about I think this is actually a good reason to host on the cloud. From a technical standpoint it's terrifying to see all these big players down at once. But what the average user likely sees is "something is wrong with the internet". So rather than seeing that your site X is down and users being angry with you, users are probably likely to think "well instagram is also down, oh and so is netflix, something big must be broken, I'll check back later" the same way users don't blame you if the power goes out.
Re: Amazon cloud outage takes down Netflix, Instagram, Pinterest, & more
#33If you are hosting on a server(as everyone is) it will, at some point fail. You have to choose a service that has minimal failure combined with quick resolution times. I think AWS fits this description...
AWS has had far more failures than my servers at any data center ever have. Running in 'the cloud', you're taking all the unavoidable points of failure (power, network, hardware) and adding in a bunch of proprietary ones (all the software that manages EC2, EBS, ELBs, internal routing between them, etc) that have all failed spectacularly at least once already with hours- to days-long resolution times.
Re: Amazon cloud outage takes down Netflix, Instagram, Pinterest, & more
#34Google, MS, Rackspace etc. ought to give a good look at all the middle layer libraries like boto, and support them to make it a matter of configuration to switch cloud service providers.
This already works well for email providers.
Re: Amazon cloud outage takes down Netflix, Instagram, Pinterest, & more
#35Earlier quoted context omitted.
Exactly. Anyone running their own data center would be subject to exactly this kind of outage, and wouldn't have access to the many, many resources that cloud providers make available to ensure successful fail-over.
Yes, I can see the "many, many resources" Amazon is making available to you in this situation. Did they send you a rubber ball that you can squeeze while waiting for your servers to come back up?
They said fail-over. Those "many, many resources" are other regions and zones.
Re: Amazon cloud outage takes down Netflix, Instagram, Pinterest, & more
#36It's funny, the more I think about I think this is actually a good reason to host on the cloud. From a technical standpoint it's terrifying to see all these big players down at once. But what the average user likely sees is "something is wrong with the internet". So rather than seeing that your site X is down and users being angry with you, users are probably likely to think "well instagram is also down, oh and so is…
Interesting thought. A couple users may be empathetic because the actual problem is somewhat visible but I'm not sure if that is an real benefit. It is of course a negative perception when they see that youtube is up and then perceive all the down sites as being less technically competent.
Re: Amazon cloud outage takes down Netflix, Instagram, Pinterest, & more
#37If you are hosting on a server(as everyone is) it will, at some point fail. You have to choose a service that has minimal failure combined with quick resolution times. I think AWS fits this description...
AWS has had far more failures than my servers at any data center ever have. Running in 'the cloud', you're taking all the unavoidable points of failure (power, network, hardware) and adding in a bunch of proprietary ones (all the software that manages EC2, EBS, ELBs, internal routing between them, etc) that have all failed spectacularly at least once already with hours- to days-long resolution times.
Yes, risk still exists, and risk profile shifts a little, but I find it to be a toward the better. Here's an anecdote:
I run applications on EC2 and RDS. I'm using Oracle. AWS has recently introduce Multi-AZ Oracle, but I haven't enabled yet. Before it was available, though, I set up a poor-man's procedure that consists of running data exports and dropping them on S3.
Now, when everything went to hell in the east, I lost an RDS instance. I couldn't do point-in-time restore, and I couldn't snapshot (both are still pending since 7 AM or so).
Luckily, I was able to spin up an RDS instance in the west, pull down the latest data from S3, and do an import. I repointed my apps at the new database, and now I'm back up.
The process took about 45 minutes. Setting up the backup scripts took about 20 minutes about 2 years ago. Now I'm just sitting on my hands waiting for the AWS ops team to fix everything. This is work I'd normally be scrambling to do myself. I'm quite happy to let those talented folks deal with it. When it's all back up and running, I'll check integrity and consistency, and I might have to restore some interim data, but for now I'm operational.
I'm sure there are worse scenarios, but the major outage last year and in the past 24-hours were quite easily mitigated.
There's something to be said for being part of a giant machine. AWS really is utility computing, so even the small guys get the benefit by virtue of standing next to the big guys.