AWS us-east-1 outage • Hacker News Board

Re: AWS us-east-1 outage

#811

Post by hn_throwaway_99 » Tue, Dec 07, 2021, 10:10 PM UTC

chucknthem wrote: post #777

uranium wrote: post #755

Earlier quoted context omitted.

Because the expected value of using AWS is greater than the expected value of self-hosting. It's not that nobody's ever heard of running on their own metal. Look back at what everyone did before AWS, and how fast they ran screaming away from it as soon as they could. Once you didn't have to do that any more, it's just so much better that the rare outages are worth it for the vast majority of startups. Medical devices…

Agree with your first point. On the second though, at some point, infrastructure like AWS are going to be more reliable than what many banks, medical device operators etc can provide themselves. asking them to stay on their own hardware is asking for that industry to remain slow, bespoke and expensive.

Hard agree with the second paragraph.

It is incredibly difficult for non-tech companies to hire quality software and infrastructure engineers - they usually pay less and the problems aren't as interesting.

Re: AWS us-east-1 outage

#812

Post by dragonwriter » Tue, Dec 07, 2021, 10:11 PM UTC

savant_penguin wrote: post #793

george3d6 wrote: post #765

Earlier quoted context omitted.

This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…

they usually beg you to use multiple availability zones though I'm not sure how many aws services are easy to spawn at multiple regions

> they usually beg you to use multiple availability zones though

Doesn't help you if it what goes down is AWS global services on which you directly, or other AWS services, depend (which tend to be tied to US-east-1).

Re: AWS us-east-1 outage

#813

Post by dylan604 » Tue, Dec 07, 2021, 10:11 PM UTC

lowbloodsugar wrote: post #733

simmanian wrote: post #715

Earlier quoted context omitted.

AWS itself has a huge single point of failure on us-east-1 region. Usually, if us-east-1 goes down, others soon follow. At that point, it doesn't matter how many regions you're deploying to.

"Usually"? When has that ever happened?

https://awsmaniac.com/aws-outages/

Re: AWS us-east-1 outage

#814

Post by WaxProlix » Tue, Dec 07, 2021, 10:12 PM UTC

naikrovek wrote: post #791

george3d6 wrote: post #765

Earlier quoted context omitted.

This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…

> AWS (and all other IAAS providers) will beg you to use multiple region will they? because AWS still puts new stuff in us-east-1 before anywhere else, and there is often a LONG delay before those things go to other regions. there are many other examples of why people use us-east-1 so often, but it all boils down to this: AWS encourage everyone to use us-east-1 and discourage the use of other regions for the same rea…

Most features roll out to IAD second, third, or fourth. PDX and CMH are good candidates for earlier feature rollout, and usually it's tested in a small region first. I use PDX (us-west-2) for almost everything these days.

I also think that they've been making a lot of the default region dropdowns and such point to CMH (us-east-2) to get folks to migrate away from IAD. Your contention that they're encouraging people to use that region just don't ring true to me.

Re: AWS us-east-1 outage

#815

Post by lp0_on_fire » Tue, Dec 07, 2021, 10:13 PM UTC

stfp wrote: post #758

boldman wrote: post #691

I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…

> tens of million dollars of profit are simply vanishing vanishing or delayed six hours? I mean

6 hours of downtime often means 6 hours of paying employees to stand around which adds up rather quickly.

Re: AWS us-east-1 outage

#816

Post by tornato7 » Tue, Dec 07, 2021, 10:13 PM UTC

nprz wrote: post #759

boldman wrote: post #691

I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…

So you're saying companies should start moving their infrastructure to the blockchain?

Ethereum has gone 5 years without a single minute of downtime, so if it's extreme reliability you're going for I don't think it can be beaten.

Re: AWS us-east-1 outage

#817

Post by i_like_waiting » Tue, Dec 07, 2021, 10:13 PM UTC

solveit wrote: post #800

imiric wrote: post #788

Earlier quoted context omitted.

> This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. > Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. That's an overblown argument. Nobody is saying that, but it's clear that businesses that maintain their ow…

This feels like a discussion that could sorely use some numbers. What are good examples of >a small business running a few websites with a few million hits per month, it might be cheaper and easier to colocate a few servers and hire a few DevOps or old-school sysadmins to administer the infrastructure. and how often do they go down?

depends I guess, I am running on-prem workstation for our DWH. So far in 2 years it went down minutes at the time, when I decided to do so, because of hardware updates. I have no idea where this narrative came from, but usually hardware you have is very reliable and doesn't turn off every 15 minutes.

Heck, I use old T430 for my home server and still it doesn't go down on completely random occasions (but thats very simplified example, I know)

Re: AWS us-east-1 outage

#818

Post by jasode » Tue, Dec 07, 2021, 10:14 PM UTC

imiric wrote: post #788

george3d6 wrote: post #765

Earlier quoted context omitted.

This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…

> This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. > Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. That's an overblown argument. Nobody is saying that, but it's clear that businesses that maintain their ow…

>, but it's clear that businesses that maintain their own infrastructure would've avoided today's AWS' outage.

When Netflix was running its own datacenters in 2008, they had a 3 day outage from a database corruption and couldn't ship DVDs to customers. That was the disaster that pushed CEO Reed Hastings to get out of managing his own datacenters and migrate to AWS.

The flaw in the reasoning that running your own hardware would avoid today's outage is that it doesn't also consider the extra unplanned outages on other days because your homegrown IT team (especially at non-tech companies) isn't as skilled as the engineers working at AWS/GCP/Azure.

Re: AWS us-east-1 outage

#819

Post by dragonwriter » Tue, Dec 07, 2021, 10:14 PM UTC

jgwil2 wrote: post #778

george3d6 wrote: post #765

Earlier quoted context omitted.

This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…

> In-house servers would lead to an insane amount of outage. That might be true, but the effects of any given outage would be felt much less widely. If Disney has an outage, I can just find a movie on Netflix to watch instead. But now if one provider goes down, it can take down everything. To me, the problem isn't the cloud per se, it's one player's dominance in the space. We've taken the inherently distributed struc…

> That might be true, but the effects of any given outage would be felt much less widely.

If my system has an hour of downtime every year and the dozen other systems it interacts with and depends on each have an hour of downtime every year, it can be better that those tend to be correlated rather than independent.

Re: AWS us-east-1 outage

#820

Post by dylan604 » Tue, Dec 07, 2021, 10:14 PM UTC

sneak wrote: post #734

dylan604 wrote: post #657

The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…

Now think of how many assets of various governments' militaries are discreetly employed as normal operational staff by FAAMG in the USA and have access to cause such events from scratch. I would imagine that the US IC (CIA/NSA) already does some free consulting for these giant companies to this end, because they are invested in that Not Being Possible (indeed, it's their job). There is a societal resilience benefit t…

> I would imagine that the US IC (CIA/NSA) already does some free consulting for these giant companies to this end,

Haha, it would be funny if the IC reaches out to BigTech when failures occur to let them know they need not be worried about data loses. They can just borrow a copy of the data IC is siphoning off them. /s?