Live data from Hacker News

AWS outage shows internet users 'at mercy' of too few providers, experts say

theguardian.com

171–180 of 223 posts

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#171
post #162

Earlier quoted context omitted.

Such a sophomoric response. It does not matter how large your storage use is exactly. The point is that nobody is going to pay to replicate that data in multiple clouds or within multiple regions of the same cloud provider. Btw, I'd love to have a link to where I could buy an SD card the size of a pinky nail that holds terabytes of data.

It absolutely matters how large your storage use is. Terabytes of storage is easily manageable on even basic consumer hardware. Terabytes of storage costs just hundreds of dollars if you are not paying the cloud tax. If you got resiliency and uptime for a extra hundred dollars a year, that would be a no-brainer for any commercial operation. The byzantine kafkaesque horror of the cloud results in trivial problems and…

[deleted]

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#172

Earlier quoted context omitted.

You duplicate the fees. But it's the same or worse trying to do multi cloud.

Which is precisely why it's not done

I don't think storage cost is the reason, more that it's hard to design for regional failures. DB by itself as one example, cross region read replica usually introduces eventual consistency to a system that'd otherwise be immediately consistent.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#173
post #126
post #116

Earlier quoted context omitted.

The "experts" in this case are > Dr Corinne Cath-Speth, the head of digital at human rights organisation Article 19 Dr. Cath-Speth has a PhD in cultural anthropology > Cori Crider, the executive director of the Future of Technology Institute A lawyer > Madeline Carr, professor of global politics and cybersecurity at University College London A professor. Her bio doesn't say what her degree is in, but she mostly seems…

[flagged]

Was that from actual experts, or bad faith strawman coverage (plenty of that about).

At least in my country, there was sober objective coverage from experts about their purpose and percentage effectiveness at reducing the range and spread of potentially infected droplets. Masks were somewhat effective for filtering incoming droplets, but most effective at containing outgoing droplets. The smaller the viral load you were exposed to the lower your chances of getting infected. Experts never claimed them to be 100% though, it was about reducing transmission rates not absolute protection.

Which is the main reason they're used in surgery too coincidentally (they aren't primarily for the surgeon's protection). Or is that an even longer running conspiracy?

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#174

This is what I call "fool's availability": reducing single points of failure (one cloud provider) without adding any actual redundancy. If you removed AWS/GCP/Azure/etc and just had 100 small providers scattered all over, the result would be hundreds of outages throughout the year, as opposed to one big outage every other year [in one region]. AWS is already way more reliable than any other provider. The real problem…

Yes but most of those companies aren't morons, they're just taking an acceptable risk. Multi-region or multi-cloud setup is nontrivial.

Most companies I've worked for (and have heard about from others) have either lacked the knowledge, or the will, to evaluate risk. They build things until they "just work", and their thought process ends there. They don't examine the design to identify its reliability and security risks. They don't calculate the losses. They still have issues, but they just happen to be acceptable most of the time.

Example1: A company's infra goes down, but it doesn't come back up correctly. People run around trying to get it working again. It takes much longer than they hoped/expected, and they lose a lot more money than they expected. This is because they never really understood the risk they were exposed to. If they understood it, they would have done more ahead of time to mitigate that much risk.

(today's outage is this case. A lot of companies are going to lose money after today, because their customers are not happy with these "acceptable risks". Presumably, losing this much money due to one outage will not be an acceptable risk in hindsight. So the company either didn't understand its risk, or it did but was too stupid to prevent it)

Example 2: A company gets hacked, and its data is either exposed or wiped. This is a much worse result; they can lose tons of money, chase off customers, damage their brand, open them up to lawsuits and fines, even tank the whole company. It's clear that this risk is pretty unacceptable. But it keeps happening. And the reason usually isn't "some genius hacker"; it was a lack of understanding the risk of not investing in security.

(there's tons of examples of these in the news. presumably, not investing in security was not an acceptable risk in hindsight when it ended their business! almost always, the people involved in making these products don't know enough about security to understand the risks. but they also don't invest in security training, mandatory security controls, checklists, processes, quality gates, etc)

You don't need multi-region or multi-cloud to mitigate reliability risks. Just like you don't need to hire a big security team or invest tons of cash to mitigate security risks. You can use your existing infra and tools, and mitigate both issues. You just have to use them wisely. It takes some effort and time, but you do it once and it pays dividends indefinitely.

Building something without identifying its security/reliability risks, and then not calculating those risks' impact, is not acceptable risk; it's ignored risk. Is tanking your company and shedding customers an acceptable risk? Well, there's one way to find out.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#175

Earlier quoted context omitted.

Containers have nothing to do with storage. They are completely orthogonal to storage (you can use Dynamo or RedShift from EC2), and many people run Docker directly on VMs. Plenty of us still spend lots of time thinking about storage and state even with containers. Containers allow me to outsource host management . I gladly spend far less time troubleshooting cloud-init, SSH, process managers, and logging/metrics age…

> Containers have nothing to do with storage. They are completely orthogonal to storage Exactly. And sure, you can use S3/Dynamo/Aurora from an EC2 box, but what would be the point of that? Just get the app running in a container, and we can look into infrastructure later. It's a very common refrain. That's why I believe Docker is strongly to linked the development of these proprietary, cloud based models of computin…

> And sure, you can use S3/Dynamo/Aurora from an EC2 box, but what would be the point of that?

An easy API? Easy replication / failover / backups? I would absolutely use S3 even with EC2.

> IAM, on the other hand, can die in dumpster fire.

I’m no great fan of AWS’s approach to IAM, but much of the pain is just the nature of fine-grained / least-privilege permissioning. On EC2 it’s more common to just grant broader permissions; IAM makes you think about least privilege, but you absolutely can grant admin for everything. And as far as a permissioning API goes, IAM is much cleaner/saner than Linux permissions.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#176

Earlier quoted context omitted.

I’m not really making a point here as much as an observation, but if my stack that I manage atop VMs in a data center goes down, my customers are pissed at me. If AWS goes down along with half the Internet, my customers are completely sympathetic.

Maybe just for you and after they realize it's part of the ongoing AWS outages, but for most folks, an outage is still their problem, and their SLA, regardless of if it's upstream from them.

I disagree. I think most customers are much more sympathetic to an AWS outage than they are to a self-managed outage. Whether that ought to be the case or not is a different question.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#177
post #131
post #125

Earlier quoted context omitted.

Do you not think it a bit too hyperbolic to throw scare quotes around experts and imply the only people who can have opinions on systemic risk are software engineers? I don't think it is unreasonable for people who haven't run or worked for a hosting service to have opinions on the policy aspect or economic impact of hyperscalers.

Anyone can have an opinion, I never said or implied otherwise. Having an opinion does not make one an expert, hence the scare quotes. The headline is misleading because when there is news about experts saying something about technology, one would naturally think that they are at least somewhat technical experts. Instead the "expert" is the director of the "Big Tech is Bad Institute" who says that "Big Tech is Bad". A…

> when there is news about experts saying something about technology, one would naturally think that they are at least somewhat technical experts.

But the experts here are not "saying something about technology". Rather they are saying something about uses of technology. So they don't need to be cloud engineers or know anything about datacenters, at all, really. What would be required (and here you may have a leg to stand on) is expertise in social and economic aspects of (now) critical infrastructure.

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#178
post #126
post #116

Earlier quoted context omitted.

The "experts" in this case are > Dr Corinne Cath-Speth, the head of digital at human rights organisation Article 19 Dr. Cath-Speth has a PhD in cultural anthropology > Cori Crider, the executive director of the Future of Technology Institute A lawyer > Madeline Carr, professor of global politics and cybersecurity at University College London A professor. Her bio doesn't say what her degree is in, but she mostly seems…

[flagged]

Those comments were made in an information regime that severly censored contrary expert opinion. We had experts in various related field who were automatically labeled as cranks simply because they disagreed with the social engineering experiment and test run of various social control mechanisms (worldwide ..).

Re: AWS outage shows internet users 'at mercy' of too few providers, experts say

#179
post #125

Earlier quoted context omitted.

Do you not think it a bit too hyperbolic to throw scare quotes around experts and imply the only people who can have opinions on systemic risk are software engineers? I don't think it is unreasonable for people who haven't run or worked for a hosting service to have opinions on the policy aspect or economic impact of hyperscalers.

> I don't think it is unreasonable for people who haven't run or worked for a hosting service to have opinions on the policy aspect or economic impact of hyperscalers. Yeah, that's completely fair. My angle was more that firstly this doesn't come across as an opinion that needs the expert in question, and secondly this is yet another case of 'Talk is cheap, show me the code', particularly when quotes in the article i…

I don't think anyone needs to produce any code. I've worked at companies with thousands of employees who don't use any cloud services.

It can be done, and contrary to marketing, it's probably cheaper and more reliable.

Post reply on HN