Live data from Hacker News

Is Amazon's cloud service too big to fail?

fnlondon.com

11–20 of 164 posts

Re: Is Amazon's cloud service too big to fail?

#11
I think about this problem every now and then for my own business, but not sure what the right answer is. Supporting multiple clouds requires more involved management of some pieces of infrastructure (e.g., DNS + healthchecks, DB replication), which introduces another point of failure.

How do people who need to have more nines of availability manage this issue with cloud providers? (EC2 and RDS promise 3.5 nines per AZ, but I imagine outages are somewhat correlated across zones)

Re: Is Amazon's cloud service too big to fail?

#12
post #6

Earlier quoted context omitted.

AWS provides a lot of features that are exclusive to their platform and can't be drop-in replaced on other providers like Azure of GCE. ELB, EFS, S3, ASGs, etc. They'd need to be replaced at the application level for other platforms. That could be a huge commitment for a decent sized system.

I don't know about ELB, EFS and ASG but: - S3 has a public protocol and many 3rd party providers support it (OpenIO, Scality, Ceph, Minio, etc), - EFS could be replaced with something like DRDB or GlusterFS, or DigitalOcean's block storage or Google Cloud's networked disks. - ELB could be replaced easily with similar services from other providers [1] if you use Kubernetes (I don't know if all have a LoadBalancer type…

One good way is to have automated tests which make sure that those rules actually work. What if all your AWS rules suddenly get deleted. How are you supposed to know if you have not forgotten any.

Re: Is Amazon's cloud service too big to fail?

#13
post #10

Is it possible for AWS to have a multi-region outage - as in is there anything connecting them that could bring them all (or several) down at once? (Apart from the result of a botched patching or update to the core software stack that was done worldwide at the same time and hopefully never happens).

No. Hence them rolling out new features region by region.

Re: Is Amazon's cloud service too big to fail?

#14
post #3

Even at a smaller scale it is a little nerve-wracking to know be so reliant on one provider. If AWS tanks there's a fair amount of code that'd need to be changed just to switch over to Azure or GCE. Failover with, e.g., email providers is easy enough, but the entire cloud stack (for lack of better terms) is a completely different ballgame.

It is one of the issues with choosing the cloud providers and taking their stack. They are hoping the cost of swapping once bought into their way is too costly to a competitor who can offer similar service cheaper. Lockin used to be considered bad but something changed with cloud providers and ops/developers don't seem to care as much anymore.

Maybe because pricing by, say Amazon, is published on their web site and therefore, the same for everyone ? Whereas before, when you were with one supplier, he could make specific price for you and leverage its position to make you pay more ? dunno...

Re: Is Amazon's cloud service too big to fail?

#15
post #3

Even at a smaller scale it is a little nerve-wracking to know be so reliant on one provider. If AWS tanks there's a fair amount of code that'd need to be changed just to switch over to Azure or GCE. Failover with, e.g., email providers is easy enough, but the entire cloud stack (for lack of better terms) is a completely different ballgame.

It is one of the issues with choosing the cloud providers and taking their stack. They are hoping the cost of swapping once bought into their way is too costly to a competitor who can offer similar service cheaper. Lockin used to be considered bad but something changed with cloud providers and ops/developers don't seem to care as much anymore.

There are some open source implementations of parts of the APIs of cloud providers that might help someone a bit when trying to migrate. For example, Minio [1] [2] implements the AWS S3 v4 API.

[1]: https://news.ycombinator.com/item?id=12392081

[2]: https://minio.io/

Re: Is Amazon's cloud service too big to fail?

#16
post #8

This does worry me. If there is a shortage of resources suddenly or a DC fire that takes out a region, then what? We have contingency against this via our own infrastructure but I worry about organisations who don't have any.

One region isn't going to be effected by fire. And AWS have dozens of regions. They're even managed as separate units by separate people. You'll notice there's never been a large, multi-region outage of AWS.

Re: Is Amazon's cloud service too big to fail?

#17
post #10

Is it possible for AWS to have a multi-region outage - as in is there anything connecting them that could bring them all (or several) down at once? (Apart from the result of a botched patching or update to the core software stack that was done worldwide at the same time and hopefully never happens).

A major solar flare and coronal mass ejection? It wouldn't just be Amazon that was affected, though.

Re: Is Amazon's cloud service too big to fail?

#18
post #2

This is (I was surprised) a pretty good article. Financial services are regulated and based on recent experience, they're concerned with systemic risk. Most industries do not have anyone responsible for worrying about this kind of thing. It seems reasonable to start worrying about the fragility potentially introduced by these massive internet infrastructure companies.

I really hate the "too big to fail" meme and I strongly agree with Bernie in that if you are too big to fail you are too big to exist.

That should be the priority.

Re: Is Amazon's cloud service too big to fail?

#19
post #2

This is (I was surprised) a pretty good article. Financial services are regulated and based on recent experience, they're concerned with systemic risk. Most industries do not have anyone responsible for worrying about this kind of thing. It seems reasonable to start worrying about the fragility potentially introduced by these massive internet infrastructure companies.

If you wanted to blow something up to make the west suffer, an AWS datacenter would probably be a pretty good target. I wonder at what point that becomes a legitimate national security concern, and the government steps in to provide protection.
Post reply on HN