Is Amazon's cloud service too big to fail? • Hacker News Board

Re: Is Amazon's cloud service too big to fail?

#121

Post by sargun » Wed, Aug 02, 2017, 7:21 PM UTC

kalleboo wrote: post #49

gaius wrote: post #43

Earlier quoted context omitted.

You can live without Netflix much longer than you can without a flushing toilet, is the point I'm making. Yet we don't have armed guards patrolling the sewageworks... It's a matter of priority how finite security personnel are deployed.

How about whatever is running on AWS GovCloud? Water outages happen often enough - they distribute water manually with tanker trucks. When sewage is overwhelmed it overflows into a river and they tell people to not go for a swim. https://en.wikipedia.org/wiki/Sanitary_sewer_overflow

GovCloud has stuff that would be bad if it goes down, but most institutions that use it have a DR plan.

Re: Is Amazon's cloud service too big to fail?

#122

Post by groby_b » Wed, Aug 02, 2017, 7:22 PM UTC

fenwick67 wrote: post #94

gaius wrote: post #43

Earlier quoted context omitted.

You can live without Netflix much longer than you can without a flushing toilet, is the point I'm making. Yet we don't have armed guards patrolling the sewageworks... It's a matter of priority how finite security personnel are deployed.

If AWS went down hard, it would be more than just "I can't watch Netflix". Some people wouldn't be able to do their computer work, send receive emails, others might not receive their paychecks or be able to pay bills.

That still ranks lower than "sewage treatment gone". Maslow's pyramid still applies.

The sewage plant hits both physiological needs and safety needs (it's a pretty dire health threat quickly). AWS has weak tendrils to financial safety, but mostly affects layers above that.

Re: Is Amazon's cloud service too big to fail?

#123

Post by martyvis » Wed, Aug 02, 2017, 7:25 PM UTC

unclebucknasty wrote: post #110

bluedino wrote: post #87

Earlier quoted context omitted.

Why blow up anything or damage any cables? Hack the computer of an Amazon employee and do your damage there. The last S4 outage was because of someone fat fingering a script, imagine what someone could do that really wanted to mess stuff up.

This. Could've saved myself a comment elsewhere on this thread. But, yeah, I'm kinda' surprised that this HN crowd in particular is so focused on hardware vectors.

And ultimately it leads you back to targeting the IT operations centre for the business where they provision equally to say both AWS and Azure for redundancy. At that point you can knock all of the biz cloud capacity off.

Re: Is Amazon's cloud service too big to fail?

#124

Post by martyvis » Wed, Aug 02, 2017, 7:25 PM UTC

unclebucknasty wrote: post #110

bluedino wrote: post #87

Earlier quoted context omitted.

Why blow up anything or damage any cables? Hack the computer of an Amazon employee and do your damage there. The last S4 outage was because of someone fat fingering a script, imagine what someone could do that really wanted to mess stuff up.

This. Could've saved myself a comment elsewhere on this thread. But, yeah, I'm kinda' surprised that this HN crowd in particular is so focused on hardware vectors.

And ultimately it leads you back to targeting the IT operations centre for the business where they provision equally to say both AWS and Azure for redundancy. At that point you can knock all of the biz cloud capacity off.

Re: Is Amazon's cloud service too big to fail?

#125

Post by nogbit » Wed, Aug 02, 2017, 7:30 PM UTC

Yes and no. By design it's not big, it just seems big. With relative RPO and RTO anyone can failover to other regions. And if you aren't leveraging multiple AZ's within a single region you need to rethink how you are using AWS.

The very nature of AWS requires Amazon to build in capabilities to handle failover. But, as they say at Amazon, "everything fails, always".

Re: Is Amazon's cloud service too big to fail?

#126

Post by galkk » Wed, Aug 02, 2017, 7:35 PM UTC

When I was working as contractor for one of big banks, which dev was concentrated on Canary Wharf, they weren't able to successfully complete disaster recovery testing on their primary database cluster for 2 years in a row, I just don't remember, was is department-wide or bank-wide.

Basically, each 6 months DR testing was failing and it was accepted as harsh reality. After seeing how they're working inside, I don't think that moving their infrastructure to AWS/Azure/Google is worst that could happen.

disc: Currently working at Amazon, but not at AWS.

Re: Is Amazon's cloud service too big to fail?

#127

Post by Kenji » Wed, Aug 02, 2017, 7:38 PM UTC

galkk wrote: post #126

When I was working as contractor for one of big banks, which dev was concentrated on Canary Wharf, they weren't able to successfully complete disaster recovery testing on their primary database cluster for 2 years in a row, I just don't remember, was is department-wide or bank-wide. Basically, each 6 months DR testing was failing and it was accepted as harsh reality. After seeing how they're working inside, I don't t…

Why did they not redo the DR testing until it worked? Normally you iterate tests and bugfixes over and over until it works. Otherwise, what's the point of the test? Being confident that your stuff does not work at all?

Re: Is Amazon's cloud service too big to fail?

#128

Post by galkk » Wed, Aug 02, 2017, 7:39 PM UTC

Kenji wrote: post #127

galkk wrote: post #126

When I was working as contractor for one of big banks, which dev was concentrated on Canary Wharf, they weren't able to successfully complete disaster recovery testing on their primary database cluster for 2 years in a row, I just don't remember, was is department-wide or bank-wide. Basically, each 6 months DR testing was failing and it was accepted as harsh reality. After seeing how they're working inside, I don't t…

Why did they not redo the DR testing until it worked? Normally you iterate tests and bugfixes over and over until it works. Otherwise, what's the point of the test? Being confident that your stuff does not work at all?

It was bank-wide activity with defined schedule etc.

Re: Is Amazon's cloud service too big to fail?

#129

Post by pavel_lishin » Wed, Aug 02, 2017, 7:48 PM UTC

martyvis wrote: post #113

It took me three reads of the first couple of paragraphs to realise that "snowball" and "snowmobile" were actually hardware products that you can touch. Tech news publishers need to do a jargon check and use appropriate punctuation, formatting or something to call out terms that 90% of readers would not have come accross

Really? It's explicitly stated in the very first paragraph, in the second sentence:

> Not the lumps of mush and ice that children chuck at each other, but Amazon’s portable information storage devices, big grey suitcases that hold huge amounts of data.

Capitalizing it might have helped, though.

Re: Is Amazon's cloud service too big to fail?

#130

Post by ianai » Wed, Aug 02, 2017, 7:54 PM UTC

groby_b wrote: post #122

fenwick67 wrote: post #94

Earlier quoted context omitted.

If AWS went down hard, it would be more than just "I can't watch Netflix". Some people wouldn't be able to do their computer work, send receive emails, others might not receive their paychecks or be able to pay bills.

That still ranks lower than "sewage treatment gone". Maslow's pyramid still applies. The sewage plant hits both physiological needs and safety needs (it's a pretty dire health threat quickly). AWS has weak tendrils to financial safety, but mostly affects layers above that.

The old historical lesson applies. People will say they're rebelling for various reasons. But at the end of the day it's for land, water, or food.