Live data from Hacker News

Is Amazon's cloud service too big to fail?

fnlondon.com

71–80 of 164 posts

Re: Is Amazon's cloud service too big to fail?

#71
post #10

Is it possible for AWS to have a multi-region outage - as in is there anything connecting them that could bring them all (or several) down at once? (Apart from the result of a botched patching or update to the core software stack that was done worldwide at the same time and hopefully never happens).

Just bringing down us-east was enough to cause quite a bit of trouble recently: https://aws.amazon.com/message/41926/

Re: Is Amazon's cloud service too big to fail?

#72
post #68

Earlier quoted context omitted.

Do you think there's a global world admin web console ?

Well as a customer how do you interact with it? Do you have one admin interface per region with its own URL or do you just go on an AWS homepage, login and control everything from there?

> Do you have one admin interface per region

This ^^^ When I go to login, it redirects me to: https://us-east-1.signin.aws.amazon.com

And for example the S3 console is on the URL https://console.aws.amazon.com/s3/home?region=us-east-1

Of course this doesn't mean I can't easily switch to a different region, or for that matter control all my regions from a single interface, but there is some distinction.

Re: Is Amazon's cloud service too big to fail?

#73
post #16
post #8

This does worry me. If there is a shortage of resources suddenly or a DC fire that takes out a region, then what? We have contingency against this via our own infrastructure but I worry about organisations who don't have any.

One region isn't going to be effected by fire. And AWS have dozens of regions. They're even managed as separate units by separate people. You'll notice there's never been a large, multi-region outage of AWS.

Yet.

Some of the traditional apps we host are vulnerable hypervisor failure be that rack, DC or region.

Re: Is Amazon's cloud service too big to fail?

#74

Earlier quoted context omitted.

One good way is to have automated tests which make sure that those rules actually work. What if all your AWS rules suddenly get deleted. How are you supposed to know if you have not forgotten any.

Terraform ( https://www.terraform.io ), which we use, is a neat way to abstract this configuration data. We keep the configuration files in git, and can do GitHub pull requests over our whole infrastructure and apply the configuration with confidence (to change the existing setup or re-create it from scratch). This works for multiple cloud providers and is great for security purposes (all changes are auditable, no co…

Essentially, a declarative configuration for infra is what you are getting at. You can take this further, using containers and orchestration tech, to abstract your application behind a declarative configuration making it infra agnostic (as it should be iMO). Obviously, not getting locked into any cloud provider services is a pre-requisite for this. Check out such an implementation here[1](full disclosure - I work here))

[1]: https://docs.hasura.io/0.14/ref/project-configuration-and-st...

Re: Is Amazon's cloud service too big to fail?

#75
post #18
post #2

This is (I was surprised) a pretty good article. Financial services are regulated and based on recent experience, they're concerned with systemic risk. Most industries do not have anyone responsible for worrying about this kind of thing. It seems reasonable to start worrying about the fragility potentially introduced by these massive internet infrastructure companies.

I really hate the "too big to fail" meme and I strongly agree with Bernie in that if you are too big to fail you are too big to exist. That should be the priority.

> I really hate the "too big to fail" meme and I strongly agree with Bernie in that if you are too big to fail you are too big to exist.

That seems like the most reasonable response. And yet, since the great recession, our policy has been "make 'too big to fail' even bigger".

The problem is that the banks have become too powerful for anyone to challenge. A Teddy Roosevelt type of political leader can't exist today.

Re: Is Amazon's cloud service too big to fail?

#76
post #43

Earlier quoted context omitted.

I feel like the point of the article we're commenting on is that AWS is far from "a few websites"

You can live without Netflix much longer than you can without a flushing toilet, is the point I'm making. Yet we don't have armed guards patrolling the sewageworks... It's a matter of priority how finite security personnel are deployed.

To the extent that this article is correct, financial services would be affected.

Re: Is Amazon's cloud service too big to fail?

#77
post #3

Even at a smaller scale it is a little nerve-wracking to know be so reliant on one provider. If AWS tanks there's a fair amount of code that'd need to be changed just to switch over to Azure or GCE. Failover with, e.g., email providers is easy enough, but the entire cloud stack (for lack of better terms) is a completely different ballgame.

It's amazing how the promise of "decentralized" internet has turned into centralized datacenters.

P2P networks, each computer being a "data store" on the internet, no one entity can control data, etc to modern day centralized cloud where a couple of players control so much.

There has been a cultural shift. In the early 2000s, the idea of storing your data somewhere else would have been weird. But now, people don't care about keeping their data on apple/google/etcs data centers.

I think it has to do with the fact that computer/internet illiterate people are now the majority whereas in the 90s/early 2000s, it was generally the computer literate on the internet.

Re: Is Amazon's cloud service too big to fail?

#79
post #62

Earlier quoted context omitted.

Why is every arrangement of characters now a "meme". That word has moved beyond devoid of meaning, at this point it's like a black hole of nothingness of a word.

> A meme (/ˈmiːm/ MEEM) is an idea, behavior, or style that spreads from person to person within a culture. Seems like it's fitting here, does it not? Certain banks being "too big to fail" is an idea passed among persons within our culture.

kind of like the idea of a meme (/ˈmiːm/ MEEM) is a meme (/ˈmiːm/ MEEM).

Re: Is Amazon's cloud service too big to fail?

#80
Hasn't anyone heard of disaster recover plans? I used to work at a medium sized insurance company and every year we had a project to update our disaster recovery plans. Including our main inhouse datacenter going down. If it was a critical system you'd better have a plan to get it back up in like 4 hours. and those were business critical we didn't have any life critical systems.
Post reply on HN