This is coming right after we switched back to AWS after trying to switch storage to Cloudflare R2. Even with this outage, I still consider AWS more reliable than Cloudflare.
AWS outage shows internet users 'at mercy' of too few providers, experts say
111–120 of 223 posts
Re: AWS outage shows internet users 'at mercy' of too few providers, experts say
#112The 'experts' also made similar criticisms with the Fastly outage in 2021 and did anything obvious change as a result? In a week's time no national newspapers will be talking about this. Meanwhile, everyone that spends actual time in these areas: - Knows that running an operation at AWS scale is difficult and any armchair critism from 'experts' is exactly that. Actions speak louder than words. - Understands that the…
I think all of that is mostly irrelevant. You don't need to pay a huge cost to avoid the small benefit, you don't need every service to be resilient to this, or any of that. You just need multiple different providers so that not everyone gets screwed at once.
But that would require companies to actually spend time and money testing and working with either a cross-provider multi-master-type system (with all the associated consistency headaches) or regularly test a functioning disaster-recovery/fallback system.
The time spent on that (let alone cost, for companies with large amounts of data) far outweighs the cost when a single region has an issue of today's scope. And you said it yourself, it's a 'small benefit'. Small benefits sound like exactly the things not worth spending time or money on.
For as much as many companies have had issues today, the daily reality is that these same companies haven't been having issues all the rest of the time (or this wouldn't have felt so shocking) and are likely to be okay with an outage of this scope (plus, everyone's too busy making noise about the issues to be working normally).
Re: AWS outage shows internet users 'at mercy' of too few providers, experts say
#113Earlier quoted context omitted.
Sounds like a pretty good shed! Like a lot of pithy commentary on the cloud, this ignores the fact the practical alternative to a shed in Virginia for most businesses is a shelf in the supply closet. "Oops, Jim Bob tripped over the power cord, guess we won't get any emails until the IT guy shows up" - this used to be a routine experience.
> "Oops, Jim Bob tripped over the power cord, guess we won't get any emails until the IT guy shows up" - this used to be a routine experience. You're not entirely wrong, but you're being hyperbolic too. I'm actually curious how old you are / how long you've worked in tech, because I started out pre-cloud and things weren't nearly as bad or as limited as you suggest. First, on-prem servers are not the only alternative…
We run a subset of our CI workload on on-prem workstations because the cost/performance ratio of consumer hardware is so much higher than servers. 1TB NVMe drive, with a 7950x/i9, 64GB RAM and gigabit networking is 100% of our failure rates with this machine have been "carpet cleaners unplugged the machine" in 2 years. Last year we had nobody in the office (due to carpet cleaning). This year we sent someone in straight after the cleaning to fix it.
Re: AWS outage shows internet users 'at mercy' of too few providers, experts say
#114Earlier quoted context omitted.
Yes but we live in a highly anti-competitive monopolized world now. With more to come under the new admin.
It’s hard to think of anything less monopolized than cloud hosting. There are hundreds of providers.
For any business that matters, your choices are amazon, google, Microsoft, and that's about it.
I couldn't even name another provider except maybe Hetzner
Re: AWS outage shows internet users 'at mercy' of too few providers, experts say
#115So, how many people will actually switch their setups to multi-cloud as a consequence of this? How many will move over to self-hosting? Or will they just do a post-incident report, wave hands around and do nothing?
Because I think it's very much the same way as it is with Cloudflare - while the large vendors aren't always openly hostile, we can just smile and hope that they don't get too keen on reminding us that they're holding us hostage.
I don't see that changing anytime soon. I've personally also used Hetzner, Contabo, Scaleway, Vultr, DigitalOcean, Time4VPS and some other platforms, but when people couple their setups to CF/AWS/GCP/Azure, typically that coupling is hard to get rid of and doing so is hard to justify.
Re: AWS outage shows internet users 'at mercy' of too few providers, experts say
#116The 'experts' also made similar criticisms with the Fastly outage in 2021 and did anything obvious change as a result? In a week's time no national newspapers will be talking about this. Meanwhile, everyone that spends actual time in these areas: - Knows that running an operation at AWS scale is difficult and any armchair critism from 'experts' is exactly that. Actions speak louder than words. - Understands that the…
The "experts" in this case are
> Dr Corinne Cath-Speth, the head of digital at human rights organisation Article 19
Dr. Cath-Speth has a PhD in cultural anthropology
> Cori Crider, the executive director of the Future of Technology Institute
A lawyer
> Madeline Carr, professor of global politics and cybersecurity at University College London
A professor. Her bio doesn't say what her degree is in, but she mostly seems to publish in political science and international relations
So, not a single technical expert. Not anyone who has ever run a hosting service before or even worked for one. Just people who write papers and sit around waiting for journalists to call them for quotes.
Re: AWS outage shows internet users 'at mercy' of too few providers, experts say
#117The whole industry walked straight into the cloud service lock-in trap. How would we begin to wind back? I also think Docker is as much to blame as the bigger cloud vendors.
I don't think it wants to. Ask any on-call engineer or support tech how they felt when, after having their phone blow up at 1am because everything is falling apart, they found out that this was an AWS-wide outage.
Re: AWS outage shows internet users 'at mercy' of too few providers, experts say
#118The 'experts' also made similar criticisms with the Fastly outage in 2021 and did anything obvious change as a result? In a week's time no national newspapers will be talking about this. Meanwhile, everyone that spends actual time in these areas: - Knows that running an operation at AWS scale is difficult and any armchair critism from 'experts' is exactly that. Actions speak louder than words. - Understands that the…
> - Knows that running an operation at AWS scale is difficult and any armchair critism from 'experts' is exactly that. Actions speak louder than words. NO. From their own reports, clearly AWS is too centralized and dependent on a specific region (us-east-1) and a specific service (DynamoDB). This has been observed for well over 10 years. Why do they stay in this centralized architecture? Cloud services need much high…
I don't deny that an incident of this scope should prompt a serious technical and process review (and as you describe it, it sounds like this is long overdue), however how often does this kind of thing not affect 2000+ services? Companies should be tracking the time they don't have issues as much as the time they do in order to actually understand if they'd be better off elsewhere.
And to be clear, I'm not at all arguing for the monopolisation of cloud providers, only stating that it's easy to point from far away and say 'This is bad' while simultaneously not doing anything to understand the cost and make that change that you say is important, because it's actually costly (in many dimensions) to do.
Re: AWS outage shows internet users 'at mercy' of too few providers, experts say
#119The only reason we can't leave AWS is because we have 500 terabytes of data in S3
What other AWS services do you depend on?
Re: AWS outage shows internet users 'at mercy' of too few providers, experts say
#120So, how many people will actually switch their setups to multi-cloud as a consequence of this? How many will move over to self-hosting? Or will they just do a post-incident report, wave hands around and do nothing? Because I think it's very much the same way as it is with Cloudflare - while the large vendors aren't always openly hostile, we can just smile and hope that they don't get too keen on reminding us that the…
GCP and Azure should be running a 10% sale/discount (Coupon code: RAINYDAY) for new accounts during the week of an AWS outage. The bean counters would take note.