Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS outage shows internet users 'at mercy' of too few providers, experts say

theguardian.com

Post reply on HN
Print view View original

191–200 of 223 posts

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#191

Post by cyberax »

maccard wrote: post #189
cyberax wrote: post #57

Earlier quoted context omitted.

You likely used a us-east-1 IAM endpoint instead of a regionalized one ( https://aws.amazon.com/blogs/security/how-to-use-regional-aw... ). We've been using it, and we're not experiencing any issues whatsoever in us-east-2. One thing that AWS should do is provide an easier way to detect these hidden dependencies. You can do that with CloudTrail if you know how to do it (filter operations by region and check that none…

We did indeed. The problem was we couldn’t log into cloud trail, or the console at all, to identify that, because IAM identity center is single region. This was a decision recommended by AWS, and blessed by our (army of) SRE teams.

But you can run TWO identity centers in different regions for the price of one(1)! IAM IDC is just a regular application hosted on the AWS infrastructure, it really has nothing special.

The hindsight is 20/20, of course, it's a good practice to audit CloudTrail periodically for unexpected regional dependencies.

(1) offer void for services that run on AWS.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#192

Post by maccard »

cyberax wrote: post #191
maccard wrote: post #189

Earlier quoted context omitted.

We did indeed. The problem was we couldn’t log into cloud trail, or the console at all, to identify that, because IAM identity center is single region. This was a decision recommended by AWS, and blessed by our (army of) SRE teams.

But you can run TWO identity centers in different regions for the price of one(1)! IAM IDC is just a regular application hosted on the AWS infrastructure, it really has nothing special. The hindsight is 20/20, of course, it's a good practice to audit CloudTrail periodically for unexpected regional dependencies. (1) offer void for services that run on AWS.

Indeed. I also noticed this morning that you're not the person I replied to, and I took your response (which was actually helpful) in the context of the original post which was "people are happy to just blame AWS when they're down".

Either way, we would have only made it one step farther in our CI, as the next step is to build a conatiner with a base image from docker hub, and that was down too. The idea of running a multi region nexus repository to avoid Docker hub outages for my 14 person engineering team seems slightly overkill!

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#193

Post by mhb »

inopinatus wrote: post #182
mhb wrote: post #126

Earlier quoted context omitted.

[flagged]

The actual experts I was paying attention to said that wearing a K/N-94/95 type mask lowers the statistical rate of transmission, that is, infection of others by your deadly virus. The subsequent findings are that cloth-type masks are less effective (but not wholly ineffective) compared to clinical/surgical masks at limiting the aerosolized viral shedding from those already infected. So if a cloth mask was all you ha…

It's unclear what you're arguing. The leading experts (Fauci/CDC) who most Americans were paying attention to were not providing this shading of meaning which you are trying to impute to them. That would be the case if they said something like N95 masks will provide excellent protection for you from the virus if worn correctly, but we have a shortage, so please make do with alternatives so that health care workers have access to them. That is not what they said. Instead they sacrificed credibility at the altar of expediency to the detriment of future trust.

What's reductive is assuming that people are motivated exclusively by self-interest instead of trusting them to make good decisions when told the truth.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#194

Post by selfhoster11 »

sunrunner wrote: post #135
kopecs wrote: post #125

Earlier quoted context omitted.

Do you not think it a bit too hyperbolic to throw scare quotes around experts and imply the only people who can have opinions on systemic risk are software engineers? I don't think it is unreasonable for people who haven't run or worked for a hosting service to have opinions on the policy aspect or economic impact of hyperscalers.

> I don't think it is unreasonable for people who haven't run or worked for a hosting service to have opinions on the policy aspect or economic impact of hyperscalers. Yeah, that's completely fair. My angle was more that firstly this doesn't come across as an opinion that needs the expert in question, and secondly this is yet another case of 'Talk is cheap, show me the code', particularly when quotes in the article i…

What code is needed to make a decision to go with a smaller provider instead of AWS?

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#195

Post by morshu9001 »

0xbadcafebee wrote: post #174
morshu9001 wrote: post #55

Earlier quoted context omitted.

Yes but most of those companies aren't morons, they're just taking an acceptable risk. Multi-region or multi-cloud setup is nontrivial.

Most companies I've worked for (and have heard about from others) have either lacked the knowledge, or the will, to evaluate risk. They build things until they "just work", and their thought process ends there. They don't examine the design to identify its reliability and security risks. They don't calculate the losses. They still have issues, but they just happen to be acceptable most of the time. Example1: A compan…

This outage would've required cross region failover to be immune to. We'll see if customers switch to whatever company was resilient, but this has happened before and the answer was no.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#196

Post by SJC_Hacker »

mhb wrote: post #126
imgabe wrote: post #116

Earlier quoted context omitted.

The "experts" in this case are > Dr Corinne Cath-Speth, the head of digital at human rights organisation Article 19 Dr. Cath-Speth has a PhD in cultural anthropology > Cori Crider, the executive director of the Future of Technology Institute A lawyer > Madeline Carr, professor of global politics and cybersecurity at University College London A professor. Her bio doesn't say what her degree is in, but she mostly seems…

[flagged]

You misunderstood then. It was mainly to protect OTHER people from a virus YOU might be carrying.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#197

Post by SJC_Hacker »

dynamite-ready wrote: post #39
Jcowell wrote: post #25

Earlier quoted context omitted.

Why is docker to blame?

It's subjective I guess, but I feel as though containerisation has greatly supported the large Cloud vendor's desire to subvert the more common model of computing... Like, before, your server was a computer, much like your desktop machine, and you programmed it much like your desktop machine. But now, people are quite happy to put their app in a Docker container and outsource all design and architecture decisions per…

Containerization was basically a way to get rid of the problem of "it works on my machine", mainly the OS version and installed libraries. Plenty of instances where program X will work on system A, but not system B, but program Y works on system B but not A. Or X is supported on Redhat/Ubuntu/etc. but you can't or don't want to build from source.

Even if that is not a problem, you avoid having to install the kitchen sink on your host and make sure everything is configured properly. Just get it working on a container, build and image and spin it up when you need it. Leaves the host machine fairly clean.

You can run a bunch of services as containers within a single host. No cloud or k8s needed. docker-compose is sufficient for testing or smallish projects.

Also, there is a security benefit because if the container is compromised, problem is limited that container not the entire host.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#198

Post by neom »

morshu9001 wrote: post #172
dylan604 wrote: post #97

Earlier quoted context omitted.

Which is precisely why it's not done

I don't think storage cost is the reason, more that it's hard to design for regional failures. DB by itself as one example, cross region read replica usually introduces eventual consistency to a system that'd otherwise be immediately consistent.

Thanks for the helpful reply! Do you think that would be still true if one accepted a constraint of the "down" version of the property served had data that was stale, say 24 hours behind what the user would have seen had they been logged in?

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#199

Post by cyberax »

maccard wrote: post #192
cyberax wrote: post #191

Earlier quoted context omitted.

But you can run TWO identity centers in different regions for the price of one(1)! IAM IDC is just a regular application hosted on the AWS infrastructure, it really has nothing special. The hindsight is 20/20, of course, it's a good practice to audit CloudTrail periodically for unexpected regional dependencies. (1) offer void for services that run on AWS.

Indeed. I also noticed this morning that you're not the person I replied to, and I took your response (which was actually helpful) in the context of the original post which was "people are happy to just blame AWS when they're down". Either way, we would have only made it one step farther in our CI, as the next step is to build a conatiner with a base image from docker hub, and that was down too. The idea of running a…

The easiest way to provide some resilience to the build process is to add a pull-through cache using AWS ECR. It might backfire due to egress costs, though, if you're building outside the AWS infrastructure.

It's actually an interesting exercise to enumerate _all_ the external dependencies. But yeah, avoiding them all seems to be less than helpful for the vast majority of users.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#200

Post by inopinatus »

mhb wrote: post #193
inopinatus wrote: post #182

Earlier quoted context omitted.

The actual experts I was paying attention to said that wearing a K/N-94/95 type mask lowers the statistical rate of transmission, that is, infection of others by your deadly virus. The subsequent findings are that cloth-type masks are less effective (but not wholly ineffective) compared to clinical/surgical masks at limiting the aerosolized viral shedding from those already infected. So if a cloth mask was all you ha…

It's unclear what you're arguing. The leading experts (Fauci/CDC) who most Americans were paying attention to were not providing this shading of meaning which you are trying to impute to them. That would be the case if they said something like N95 masks will provide excellent protection for you from the virus if worn correctly, but we have a shortage, so please make do with alternatives so that health care workers ha…

This information was and is widely available.

Your earlier statement was entirely framed in self-interest, so you don’t get to complain about being pulled up on that now.

Post reply on HN