Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS outage shows internet users 'at mercy' of too few providers, experts say

theguardian.com

Post reply on HN
Print view View original

71–80 of 223 posts

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#71

Post by jraph »

racl101 wrote: post #11

I find it weird many people are just realizing this. I've had this conversation with regards to talking about what should happen if a couple of bad earth quakes, not even "the big one", were to occur. But on the other hand, maybe I hang around too many tech people to not empathically understand the other point of view.

I think many (most?) non tech people don't even know that Amazon is first and foremost a cloud provider (and one of the biggest at that, if not the biggest) and that its market thing is almost a side activity at this point.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#72

Post by swiftcoder »

TZubiri wrote: post #50
impure wrote: post #29

We already have diversification. You can rent a VPS from hundreds of possible companies. And people are very happy with them, it seems every month or two there’s a post here about how some company slashed their cloud bill by switching to a VPS. What we have here is a lock-in and marketing problem.

Amazon offers VPS as well, EC2 instances, were those affected? I think they weren't.

Our actual running instances were pretty much fine throughout, as was the RDS cluster, but we had no way to launch new instances (or auto-scale), and no way to invoke any of the other AWS services (IAM, SQS, Lambda, etc). Also no cloud watch logs/metrics for the duration, so limited visibility.

Overall not that bad for us, but if you had more high-level service dependencies, there would have been impact.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#73

Post by toast0 »

neom wrote: post #5

Been a while since I worked in cloud but at least when I got out of it, the primitives where all shoring up to be generally very similar. Did multi cloud redundancy end up being too expensive? Tech didn't line up enough? No good business case? The elastic cloud story that never was? https://www.slideshare.net/slideshow/pets-vs-cattle-the-elas... What happened?

It seems that clouds balance their budget on egress charges... which leads to cross cloud communication being too expensive to setup multi cloud redundancy. Cross region redundancy is often too expensive too. Even cross availability zones is too expensive for some clouds and applications. (Cross region redundancy in a single cloud doesn't always work out, if the cloud has an outage on a global subsystem, or the broken subsystem gets pushed to multiple regions before exhibiting symptoms)

Additionally, moving your load to a different cloud can be challenging while one is down. It ends up being a lot of work that pays off for a few hours a year. For a lot of applications, it's better to just suffer the downtime and spend money on other things.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#74

Post by swiftcoder »

interroboink wrote: post #56
aurumque wrote: post #41

Earlier quoted context omitted.

For those of us who have been using AWS for almost 20 years now, I can't imagine why anyone would willingly choose us-east-1 for anything. It is the oldest, highest traffic, most critical path region and is subject to turbulence.

By some logic, that would mean it is the most battle-tested and highest-stakes (and therefore most carefully-managed) choice. I.e. reasons in favor. Not that I disagree with you, but maybe not for the reasons you say (:

> By some logic, that would mean it is the most battle-tested and highest-stakes (and therefore most carefully-managed) choice

As someone who used to work on the inside, us-east-1 has the biggest pile of legacy workarounds for internal AWS issues, it has a variety of legacy API behaviours that don't exist in other regions, and because everyone picks it as the default, it has significantly more pressure on contested resources (i.e. things like spot instance pools).

Plus since it's the default in all the tooling, if you ever decide to go multi-region, you'll find tons of things break right away.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#76

Post by jewel »

greenavocado wrote: post #69

The only reason we can't leave AWS is because we have 500 terabytes of data in S3

Talk to the other vendors. I know of a place that had about that same amount and decided to have a redundant copy of all of their data in another vendor's S3-compatible product. That vendor paid for all of their egress fees as long as they signed a 12-month contract and used their tool for the migration.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#77

Post by justapassenger »

neom wrote: post #5

Been a while since I worked in cloud but at least when I got out of it, the primitives where all shoring up to be generally very similar. Did multi cloud redundancy end up being too expensive? Tech didn't line up enough? No good business case? The elastic cloud story that never was? https://www.slideshare.net/slideshow/pets-vs-cattle-the-elas... What happened?

There’s a huge difference between “similar” and “works and is ROI positive for my business across the whole lifecycle”.

Multi cloud redundancy is like Java being a solution to platform independency.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#78

Post by dylan604 »

TZubiri wrote: post #60
dylan604 wrote: post #45

Earlier quoted context omitted.

Cross region isn't simple when you have terabytes of storage in buckets in a region. Building services in other regions without that data doesn't really do any good. Maintaining instances in various regions is easy, but it's that data that complicates everything. If you need to use the instances in a different region because your main region is down, you still can't do anything because those cross region instances ca…

Bottomline is that AWS gives you the tools to survive this outage within their own ecosystem. If there's an issue with relying only on AWS it has not been expressed in this outage.

exactly what tools helps make your large volume of data stored in a down region available to other regions without duplicating the monthly storage fees?

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#79

Post by dingnuts »

tlogan wrote: post #61
aurumque wrote: post #41

Earlier quoted context omitted.

For those of us who have been using AWS for almost 20 years now, I can't imagine why anyone would willingly choose us-east-1 for anything. It is the oldest, highest traffic, most critical path region and is subject to turbulence.

I think it is a little complicated. For example, your service might be using full failover but you use API from other service which are down. Or you might use BART to come to work and you got stuck: https://www.kqed.org/news/12060687/bart-resumes-service-but-...

ha! I saw another comment on here talking about how ec2 doesn't need to be held to the same standard as the power company because it's not as important as real infrastructure.

wish I'd already had this link in my back pocket. our industry needs to take its job, as a whole, much more seriously.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#80

Post by sunrunner »

The 'experts' also made similar criticisms with the Fastly outage in 2021 and did anything obvious change as a result? In a week's time no national newspapers will be talking about this.

Meanwhile, everyone that spends actual time in these areas:

- Knows that running an operation at AWS scale is difficult and any armchair critism from 'experts' is exactly that. Actions speak louder than words.

- Understands that the cost of actually accounting for this kind of scenarios is incredibly high for the benefit in most cases

- Knows that genuinely 'critical' services (i.e. health) should be designed to account for this, and every other 'serious' issue such as 'I can't log in to Fortnite' just shows what the price and effort of actually making that work is versus how much it costs affected companies when it happens

- Knows how much time national newspapers spend actually talking about the importance of multi-region/multi-cloud redundancy, that is, it's zero until the one day where it happens and then it's old news

- Is just curious as to just what exactly happened from a technical perspective

This isn't to say that good blameless post-mortem shouldn't happen to figure out process and technical issues, but the armchair criticism with no actual followup? All noise, no signal.

Post reply on HN