AWS outage shows internet users 'at mercy' of too few providers, experts say • Hacker News Board

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#121

Post by wagwang » Mon, Oct 20, 2025, 8:15 PM UTC

imgabe wrote: post #116

sunrunner wrote: post #80

The 'experts' also made similar criticisms with the Fastly outage in 2021 and did anything obvious change as a result? In a week's time no national newspapers will be talking about this. Meanwhile, everyone that spends actual time in these areas: - Knows that running an operation at AWS scale is difficult and any armchair critism from 'experts' is exactly that. Actions speak louder than words. - Understands that the…

The "experts" in this case are > Dr Corinne Cath-Speth, the head of digital at human rights organisation Article 19 Dr. Cath-Speth has a PhD in cultural anthropology > Cori Crider, the executive director of the Future of Technology Institute A lawyer > Madeline Carr, professor of global politics and cybersecurity at University College London A professor. Her bio doesn't say what her degree is in, but she mostly seems…

Opinions are valid but also worthless. Just give me a funny tweet to digest the situation.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#122

Post by godelski » Mon, Oct 20, 2025, 8:16 PM UTC

dynamite-ready wrote: post #39

Jcowell wrote: post #25

Earlier quoted context omitted.

Why is docker to blame?

It's subjective I guess, but I feel as though containerisation has greatly supported the large Cloud vendor's desire to subvert the more common model of computing... Like, before, your server was a computer, much like your desktop machine, and you programmed it much like your desktop machine. But now, people are quite happy to put their app in a Docker container and outsource all design and architecture decisions per…

Containers let me essentially build those machines but at the actual requirements I need for a particular system. So instead of 10 machines I can build 1. I then don't need to upgrade that machine if my service changes.

Its also more resilient because I can trash a container and load up a new one with low overhead. I can't really do that with a full machine. It also gives some more security by sandboxing.

This does lead to laziness by programmers accelerated by myopic management. "It works" except when it doesn't. Easy to say you just need to restart the container then to figure out the actual issue.

But I'm not sure what that has to do with cloud. You'd do the same thing self hosting. Probably save money too. Though I'm frequently confused why people don't do both. Self host and host in the cloud. That's how you create resilience. Though you also need to fix problems rather than restart to be resilient too.

I feel like our industry wants to move fast but without direction. It's like we know velocity matters but since it's easier to read the speedometer we pretend they're the same thing. So fast and slow makes sense. Fast by magnitude of the vector. Slow if you're measuring how fast we make progress in the intended direction.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#123

Post by zenoprax » Mon, Oct 20, 2025, 8:16 PM UTC

sunrunner wrote: post #80

The 'experts' also made similar criticisms with the Fastly outage in 2021 and did anything obvious change as a result? In a week's time no national newspapers will be talking about this. Meanwhile, everyone that spends actual time in these areas: - Knows that running an operation at AWS scale is difficult and any armchair critism from 'experts' is exactly that. Actions speak louder than words. - Understands that the…

I think your third point is what I've had to attune to when criticizing cloud dependence. I think if your entire source of revenue is dependent on AWS then you should be prepared for 16+ hours of downtime per year. Individuals notice it more when something is down for hours but with good observability I am guessing the business notices it more when performance drags for the other 8742 hours of the year. Bursts of downtime per day can still be attributed to the device, wifi, ISP, or some other intermediary's DNS/BGP.

If your margins are so tight that 16 hours of downtime will bankrupt you then I think either: a) I have no idea how to run a business; or b) you have no idea how to run a business. I'm also biased because I love highly fault-tolerant, geo-redundant, durable systems much more than "good enough for this KPI".

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#124

Post by xp84 » Mon, Oct 20, 2025, 8:17 PM UTC

In 2011 there was some kind of big outage at some major AWS US-east pop. I started a job at a company (very boring B2C startup) which had taken the lesson from that, that "cloud anything is dangerous."

They went and bought a bunch of literal servers and installed them in a datacenter, 90 miles away from our offices, and this is where all our applications ran for the remainder of that company's existence (about 6 more years). For the whole time I was at that company, we had somewhat more, and usually more lengthy, outages than the average startup. The only difference is that when some piece of networking gear took a crap, or a disk failed, or whatever, our guys had to diagnose and resolve it (Their karma, I guess, since this was their idea).

Anyway, I do think it would be good if at least so-calld 'tech companies' had a little less obsession to outsource everything -- even easy things -- to AWS, GCP, and Azure. I feel that way mainly for cost reasons as many of these services are wildly overpriced. But also we shouldn't kid ourselves by ignoring the advantages of operating at the scale those guys do. They can afford to have multiple absolute wizards available around the clock who make sure that when a problem happens, it's not the kind of "S-show" we had at my old company where we're all on a slack room or zoom or whatever and just guessing at to try for half an hour before we can figure out what the actual issue is.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#125

Post by kopecs » Mon, Oct 20, 2025, 8:17 PM UTC

imgabe wrote: post #116

sunrunner wrote: post #80

The 'experts' also made similar criticisms with the Fastly outage in 2021 and did anything obvious change as a result? In a week's time no national newspapers will be talking about this. Meanwhile, everyone that spends actual time in these areas: - Knows that running an operation at AWS scale is difficult and any armchair critism from 'experts' is exactly that. Actions speak louder than words. - Understands that the…

The "experts" in this case are > Dr Corinne Cath-Speth, the head of digital at human rights organisation Article 19 Dr. Cath-Speth has a PhD in cultural anthropology > Cori Crider, the executive director of the Future of Technology Institute A lawyer > Madeline Carr, professor of global politics and cybersecurity at University College London A professor. Her bio doesn't say what her degree is in, but she mostly seems…

Do you not think it a bit too hyperbolic to throw scare quotes around experts and imply the only people who can have opinions on systemic risk are software engineers? I don't think it is unreasonable for people who haven't run or worked for a hosting service to have opinions on the policy aspect or economic impact of hyperscalers.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#126

Post by mhb » Mon, Oct 20, 2025, 8:18 PM UTC

imgabe wrote: post #116

sunrunner wrote: post #80

The 'experts' also made similar criticisms with the Fastly outage in 2021 and did anything obvious change as a result? In a week's time no national newspapers will be talking about this. Meanwhile, everyone that spends actual time in these areas: - Knows that running an operation at AWS scale is difficult and any armchair critism from 'experts' is exactly that. Actions speak louder than words. - Understands that the…

The "experts" in this case are > Dr Corinne Cath-Speth, the head of digital at human rights organisation Article 19 Dr. Cath-Speth has a PhD in cultural anthropology > Cori Crider, the executive director of the Future of Technology Institute A lawyer > Madeline Carr, professor of global politics and cybersecurity at University College London A professor. Her bio doesn't say what her degree is in, but she mostly seems…

[flagged]

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#127

Post by franz_vlkshp » Mon, Oct 20, 2025, 8:18 PM UTC

SpicyLemonZest wrote: post #84

labrador wrote: post #68

Kieran Healy @kjhealy@mastodon.social Always worth taking sentences that use “the Cloud” or “the Internet” and try replacing those phrases with “A shed in Virginia” to see how they hold up. “Our service is fully based in a shed in Virginia”; “All my files are in a shed in Virginia”; “A shed in Virginia was designed to survive a nuclear war”, etc. https://mastodon.social/@kjhealy/115407725852594322

Sounds like a pretty good shed! Like a lot of pithy commentary on the cloud, this ignores the fact the practical alternative to a shed in Virginia for most businesses is a shelf in the supply closet. "Oops, Jim Bob tripped over the power cord, guess we won't get any emails until the IT guy shows up" - this used to be a routine experience.

on the other hand, that's a small price to pay to having total control and physical access to your own infrastructure. if the sysadmin did his job properly, an incident like that shouldn't require anything else but to plug the server back in and hit the power switch. but then if he did his job properly, no one but IT should be tripping on power cables to begin with.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#128

Post by inopinatus » Mon, Oct 20, 2025, 8:19 PM UTC

alecco wrote: post #109

sunrunner wrote: post #80

The 'experts' also made similar criticisms with the Fastly outage in 2021 and did anything obvious change as a result? In a week's time no national newspapers will be talking about this. Meanwhile, everyone that spends actual time in these areas: - Knows that running an operation at AWS scale is difficult and any armchair critism from 'experts' is exactly that. Actions speak louder than words. - Understands that the…

> - Knows that running an operation at AWS scale is difficult and any armchair critism from 'experts' is exactly that. Actions speak louder than words. NO. From their own reports, clearly AWS is too centralized and dependent on a specific region (us-east-1) and a specific service (DynamoDB). This has been observed for well over 10 years. Why do they stay in this centralized architecture? Cloud services need much high…

Even wearing my ex-AWS hat and understanding to some degree the internal complexity of these services, I too am boggled that foundational stuff is still out of Virginia and not a separately operated global region for the subset of control-plane dependencies that can’t be refactored into tolerating eventual consistency (such as parts of IAM).

We always used to talk a lot about minimising blast radius and there’s been enough time, and enough scale, to fix it.

Nevertheless the Guardian’s choice to label self-promoting policy wonks as “experts” is a cringe-inducing reminder that journalists don’t know anything about anything.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#129

Post by greenavocado » Mon, Oct 20, 2025, 8:19 PM UTC

ovaistariq wrote: post #119

greenavocado wrote: post #69

The only reason we can't leave AWS is because we have 500 terabytes of data in S3

What other AWS services do you depend on?

Mostly EC2 for data mining terabytes of historical data stored in S3. Production usage is fairly lightweight compared to the EC2 and S3 stuff. We did cut our bill a lot by moving to single AZ redundancy.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#130

Post by midtake » Mon, Oct 20, 2025, 8:21 PM UTC

There are many public clouds and VPS providers out there. Who the fuck are these experts?

The real issue is that business pricks will cut costs and single-homing in a single availability zone will be the only workable solution.

On top of that, infrastructure ops are seen as a nuisance who get in the way of the sexy stuff like shipping your latest code changes now. If you complicate the ops pipeline that gets in the way of sexy dev work. So fuck that just ship lol!