Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is Amazon's cloud service too big to fail?

fnlondon.com

Post reply on HN
Print view View original

101–110 of 164 posts

Re: Is Amazon's cloud service too big to fail?

#101

Post by plandis »

cm2187 wrote: post #67
mediascreen wrote: post #45

Earlier quoted context omitted.

Well, you need to disable redundant ISP or power cables to all datacenters in a region. and that would probably be pretty easy to recover from in a day or two. I imagine Amazon has some on site security measures as well. It would be interesting to see how important services would cope with their main region going down for more than a few days.

Not that I think it is wise to discuss optimising a terrorist activity on a public forum, though it is interesting from a threat analysis point of view. You don't need to severe both power and data cables. For power the datacentre should be able to cope for a while, particularly if it has access to fuel deliveries. Data cable should be both easier to severe and more difficult to recover from (not the least to identif…

But if that's your methodology, you'd need to cut several cables in different locations simultaneously. A single cable doesn't take too long to fix (a couple hours, tops) and there are probably several redundant backups to handle bandwidth in the interim.

Re: Is Amazon's cloud service too big to fail?

#102

Post by kevan »

patta54 wrote: post #93
barsonme wrote: post #3

Even at a smaller scale it is a little nerve-wracking to know be so reliant on one provider. If AWS tanks there's a fair amount of code that'd need to be changed just to switch over to Azure or GCE. Failover with, e.g., email providers is easy enough, but the entire cloud stack (for lack of better terms) is a completely different ballgame.

I warn other developers at my company about this. When new projects spin up they're often very excited about using new Amazon services and will make any excuse to choose an AWS product over a stable open source solution. If I were a manager, I'd be very worried over the vendor lock-in. I don't understand the preference for AWS over open source in many cases. Their services are "reliable", but they often have minute r…

>Why use SNS/SQS when there are free pubsub/message buses out there?

Free is never really free. There's always a tradeoff in engineering time and money when you choose to run your own stack instead of paying to use a stable, well-established service. Oftentimes running your own will be cheaper overall, but you have to do that cost-benefit comparison for yourself.

Re: Is Amazon's cloud service too big to fail?

#103

Post by kirykl »

awkwarddaturtle wrote: post #77
barsonme wrote: post #3

Even at a smaller scale it is a little nerve-wracking to know be so reliant on one provider. If AWS tanks there's a fair amount of code that'd need to be changed just to switch over to Azure or GCE. Failover with, e.g., email providers is easy enough, but the entire cloud stack (for lack of better terms) is a completely different ballgame.

It's amazing how the promise of "decentralized" internet has turned into centralized datacenters. P2P networks, each computer being a "data store" on the internet, no one entity can control data, etc to modern day centralized cloud where a couple of players control so much. There has been a cultural shift. In the early 2000s, the idea of storing your data somewhere else would have been weird. But now, people don't ca…

I was pretty befuddled when my company IT switched from self hosted storage to commercial cloud accounts for incredibly sensitive info.

I think the reasoning was cloud accounts are easier for the masses than mapping a drive and accessing over VPN

Re: Is Amazon's cloud service too big to fail?

#104

Post by plandis »

patta54 wrote: post #93
barsonme wrote: post #3

Even at a smaller scale it is a little nerve-wracking to know be so reliant on one provider. If AWS tanks there's a fair amount of code that'd need to be changed just to switch over to Azure or GCE. Failover with, e.g., email providers is easy enough, but the entire cloud stack (for lack of better terms) is a completely different ballgame.

I warn other developers at my company about this. When new projects spin up they're often very excited about using new Amazon services and will make any excuse to choose an AWS product over a stable open source solution. If I were a manager, I'd be very worried over the vendor lock-in. I don't understand the preference for AWS over open source in many cases. Their services are "reliable", but they often have minute r…

You're also forgetting that if you set up something on your own you also have all the hardware concerns as well. You need to procure hosts, provision them properly, deploy them, monitor them, scale them, fix them. That infrastructure cost doesn't go to zero but it is significantly reduced using a cloud provider.

Re: Is Amazon's cloud service too big to fail?

#105

Post by jpalomaki »

This goes to beyond having a plan-B for hosting your own stuff somewhere else. Think about all the 3rd party services you are depending on. Then think about how many dependencies those services have. How many trace back to Amazon on some level?

The connections that could cause problems may not be obvious. For example network provider running into trouble because a ticketing or monitoring system that depends Amazon does not work. Hardware supplier not being able to ship spare parts for your on-premise SAN because logistics company runs into trouble due to issues at Amazon.

Re: Is Amazon's cloud service too big to fail?

#106

Post by unclebucknasty »

myusernameisok wrote: post #95
BurningFrog wrote: post #83

Earlier quoted context omitted.

Knocking out all of AWS is very different from knocking out a single data center.

Especially because AWS regions are broken up into multiple availability zones (data centres in the same area). So taking out a single data centre won't do much if the AWS customers have correctly designed their systems for high-availability (ie having redundant instances in other AZs/regions with their data backed up elsewhere).

This all assumes that "taking out" datacenters is a physical/hardware operation.

When you widen the potential attack surface to include software vulnerabilities, unauthorized access, process flaws and other "soft" vectors, a much wider--possibly coordinated--attack that is potentially far more crippling can be imagined.

Re: Is Amazon's cloud service too big to fail?

#107

Post by plandis »

askvictor wrote: post #31
smegel wrote: post #10

Is it possible for AWS to have a multi-region outage - as in is there anything connecting them that could bring them all (or several) down at once? (Apart from the result of a botched patching or update to the core software stack that was done worldwide at the same time and hopefully never happens).

A cascading electrical grid failure? I don't know if there are any interconnects between the regions with the DC's, but if there were that might be a concern. Though at that stage, presumably most of the US is without power, hence not so much need for AWS.

I think each DC has at least two power sources and probably a backup generator. I think that's why cloud providers have been so reluctant to open in Africa, diversified power is apparently a problem.

Re: Is Amazon's cloud service too big to fail?

#108

Post by unclebucknasty »

fenwick67 wrote: post #94
gaius wrote: post #43

Earlier quoted context omitted.

You can live without Netflix much longer than you can without a flushing toilet, is the point I'm making. Yet we don't have armed guards patrolling the sewageworks... It's a matter of priority how finite security personnel are deployed.

If AWS went down hard, it would be more than just "I can't watch Netflix". Some people wouldn't be able to do their computer work, send receive emails, others might not receive their paychecks or be able to pay bills.

>would be more than just "I can't watch Netflix".

And, I think even more than we can imagine. It's one thing to count the number of services that are direct customers/dependents of AWS, and we really don't know how deep that goes.

But, add to that the non-AWS based services that directly or indirectly rely on AWS-based services.

Re: Is Amazon's cloud service too big to fail?

#109

Post by patta54 »

plandis wrote: post #104
patta54 wrote: post #93

Earlier quoted context omitted.

I warn other developers at my company about this. When new projects spin up they're often very excited about using new Amazon services and will make any excuse to choose an AWS product over a stable open source solution. If I were a manager, I'd be very worried over the vendor lock-in. I don't understand the preference for AWS over open source in many cases. Their services are "reliable", but they often have minute r…

You're also forgetting that if you set up something on your own you also have all the hardware concerns as well. You need to procure hosts, provision them properly, deploy them, monitor them, scale them, fix them. That infrastructure cost doesn't go to zero but it is significantly reduced using a cloud provider.

I'm not arguing against cloud platforms in general; just the irrational use of very specialized services they offer. I can run a containerized service that uses open source packages on any of the cloud computing platforms. Now if I used Athena, SQS/SNS, DynamoDB, ELB, Lambda, EC2 that would make me very nervous, and I see other devs designing these stacks all the time. I guess I shouldn't care as much, because I'm not going to be the one to migrate that when the company gets a better deal from another platform service.

Re: Is Amazon's cloud service too big to fail?

#110

Post by unclebucknasty »

bluedino wrote: post #87
cm2187 wrote: post #30

Earlier quoted context omitted.

All you need to blow is a few cables.

Why blow up anything or damage any cables? Hack the computer of an Amazon employee and do your damage there. The last S4 outage was because of someone fat fingering a script, imagine what someone could do that really wanted to mess stuff up.

This. Could've saved myself a comment elsewhere on this thread.

But, yeah, I'm kinda' surprised that this HN crowd in particular is so focused on hardware vectors.

Post reply on HN