Is Amazon's cloud service too big to fail? • Hacker News Board

Re: Is Amazon's cloud service too big to fail?

#51

Post by smegel » Wed, Aug 02, 2017, 10:06 AM UTC

askvictor wrote: post #31

smegel wrote: post #10

Is it possible for AWS to have a multi-region outage - as in is there anything connecting them that could bring them all (or several) down at once? (Apart from the result of a botched patching or update to the core software stack that was done worldwide at the same time and hopefully never happens).

A cascading electrical grid failure? I don't know if there are any interconnects between the regions with the DC's, but if there were that might be a concern. Though at that stage, presumably most of the US is without power, hence not so much need for AWS.

Well I guess a nationwide power outage will have bigger implications than Netflix going down...

Re: Is Amazon's cloud service too big to fail?

#52

Post by gaius » Wed, Aug 02, 2017, 10:06 AM UTC

icebraining wrote: post #48

gaius wrote: post #40

Earlier quoted context omitted.

But in an ideal world you wouldn't tie your application to a specific cloud provider. You can do that easily if you just treat clouds merely as hosted hypervisors and think entirely in terms of VMDKs. But this doesn't make commercial sense to do at least in the short term - you need to utilise the layered services you are paying for anyway or you might as well just run your own DC.

It still makes sense for its elastic properties (from which EC2 got its name). You can't rent half a DC for an hour, but you can spawn generic instances from VMDKs on different providers with a fairly small abstraction layer.

Your data still needs to live somewhere and giant VMDKs being copied around aren't a reasonable solution, I'd argue.

Re: Is Amazon's cloud service too big to fail?

#53

Post by mgkimsal » Wed, Aug 02, 2017, 10:22 AM UTC

gaius wrote: post #43

peteretep wrote: post #38

Earlier quoted context omitted.

I feel like the point of the article we're commenting on is that AWS is far from "a few websites"

You can live without Netflix much longer than you can without a flushing toilet, is the point I'm making. Yet we don't have armed guards patrolling the sewageworks... It's a matter of priority how finite security personnel are deployed.

"Yet we don't have armed guards patrolling the sewageworks"

There's probably a moderate amount of unseen security in larger areas surrounding water/sewage plants.

Also, we've dealt with regional natural disasters before. Drought, fire, flooding, etc - we have experience mobilizing resources and redistributing as needed.

We've never dealt with an extended outage of one or more AWS data centers for days at a time. How many govt/university systems would be unable to function because of direct or indirect dependencies on AWS-related services?

S3 going down for, let's say 4 days, would cause big havoc on so many projects and systems I know of.

I'm pretty sure most people have no clue how much of their data and systems functionality is reliant on AWS-related services.

Re: Is Amazon's cloud service too big to fail?

#54

Post by empath75 » Wed, Aug 02, 2017, 10:28 AM UTC

peteretep wrote: post #19

dalbasal wrote: post #2

This is (I was surprised) a pretty good article. Financial services are regulated and based on recent experience, they're concerned with systemic risk. Most industries do not have anyone responsible for worrying about this kind of thing. It seems reasonable to start worrying about the fragility potentially introduced by these massive internet infrastructure companies.

If you wanted to blow something up to make the west suffer, an AWS datacenter would probably be a pretty good target. I wonder at what point that becomes a legitimate national security concern, and the government steps in to provide protection.

Most people wouldn't even notice. They'd have to blow up a region and at that point, you have bigger problems.

Re: Is Amazon's cloud service too big to fail?

#55

Post by Rjevski » Wed, Aug 02, 2017, 10:28 AM UTC

smegel wrote: post #10

Is it possible for AWS to have a multi-region outage - as in is there anything connecting them that could bring them all (or several) down at once? (Apart from the result of a botched patching or update to the core software stack that was done worldwide at the same time and hopefully never happens).

Well given that you can manage services across all regions from a single web interface, I assume someone compromising this web interface would be able to control and bring stuff down across all regions.

Re: Is Amazon's cloud service too big to fail?

#56

Post by jrimbault » Wed, Aug 02, 2017, 10:38 AM UTC

wiz21c wrote: post #14

PaulKeeble wrote: post #5

Earlier quoted context omitted.

It is one of the issues with choosing the cloud providers and taking their stack. They are hoping the cost of swapping once bought into their way is too costly to a competitor who can offer similar service cheaper. Lockin used to be considered bad but something changed with cloud providers and ops/developers don't seem to care as much anymore.

Maybe because pricing by, say Amazon, is published on their web site and therefore, the same for everyone ? Whereas before, when you were with one supplier, he could make specific price for you and leverage its position to make you pay more ? dunno...

They could show everyone different prices (like plane tickets)

Re: Is Amazon's cloud service too big to fail?

#57

Post by michaelt » Wed, Aug 02, 2017, 10:40 AM UTC

arethuza wrote: post #33

wiz21c wrote: post #14

Earlier quoted context omitted.

Maybe because pricing by, say Amazon, is published on their web site and therefore, the same for everyone ? Whereas before, when you were with one supplier, he could make specific price for you and leverage its position to make you pay more ? dunno...

I'd be very surprised if big users of AWS or Azure pay the rates advertised on the public web sites.

Right, nobody pays /more/ than the published prices.

And prospective users can look at the published prices, and see that historically they've gone down more than they've gone up (although obviously that trend could reverse).

So people think they're safe from the risk of Amazon quadrupling their bill over night.

Of course, vendor lock-in can have other negative effects, but apparently people aren't worried about them, or at least think AWS is no worse than the alternatives.

Re: Is Amazon's cloud service too big to fail?

#58

Post by sokoloff » Wed, Aug 02, 2017, 10:42 AM UTC

latch wrote: post #46

arethuza wrote: post #33

Earlier quoted context omitted.

I'd be very surprised if big users of AWS or Azure pay the rates advertised on the public web sites.

Like most large business, AWS has a sales department and sales engineers that behave like you expect. Rates are absolutely negotiable.

Having negotiated both, AWS is negotiable within a range that is much tighter than the range for enterprise sales of Cisco, EMC, etc gear. (Or AWS has better negotiators, but I've never gotten a call "Hey, Qx is about to end and I need to hit my numbers, so is there anything we can pull forward" from an AWS rep.)

Re: Is Amazon's cloud service too big to fail?

#59

Post by dantiberian » Wed, Aug 02, 2017, 10:43 AM UTC

smegel wrote: post #10

Is it possible for AWS to have a multi-region outage - as in is there anything connecting them that could bring them all (or several) down at once? (Apart from the result of a botched patching or update to the core software stack that was done worldwide at the same time and hopefully never happens).

My memory may be wrong, but I thought several regions were affected by the recent S3 outage? Also, I suspect that if us-east-1 went down completely that that would have a debilitating effect on the others.

Re: Is Amazon's cloud service too big to fail?

#60

Post by sokoloff » Wed, Aug 02, 2017, 10:49 AM UTC

fovc wrote: post #11

I think about this problem every now and then for my own business, but not sure what the right answer is. Supporting multiple clouds requires more involved management of some pieces of infrastructure (e.g., DNS + healthchecks, DB replication), which introduces another point of failure. How do people who need to have more nines of availability manage this issue with cloud providers? (EC2 and RDS promise 3.5 nines per…

Very, very few businesses should be architecting to ensure higher than 99.95% availability, IMO. (Less than 4.5 hours of downtime per year.)

Users are patient enough to give you a pass if you're down that amount (especially if you're down that amount while 1/3rd of the internet is also down).

Our largest e-commerce retail site does over $1BB/yr in fairly high-margin sales and still targets "only" 99.95% availability (generally it exceeds that with actual results, but we don't target higher than that). It's a hybrid of on-prem and cloud services backing that, migrating towards the cloud, but will never be 100% cloud as we own and run factories with on-prem equipment.

(I know you asked "how" and I answered "whether", but I thought it relevant.)