Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

641–646 of 646 posts

Re: Tell HN: AWS appears to be down again

#641

Post by newobj »

hinkley wrote: post #486
kalleth wrote: post #277

Earlier quoted context omitted.

I'd be surprised if they needed backups for a few hours of downtime with (reportedly) complete recovery where no data was corrupted. There are industries where this would be required, and it's possible I guess, but neither of these downtime events were "data loss" events, just availability events for short-ish periods of time that wouldn't - for me - result in activating our DR plans. I must admit that I do always tr…

AWS has had at least one documented incident where a region had an S3 failure that was not recoverable. They lost about 2% of all data. That might not sound like much but if you have a lot of data, partial restoration of that data doesn't necessarily leave your system in a functional state. If it loses my compiled CSS files I might be able to redeploy my app to fix it. Then again if I'm a SaaS company and that file w…

Sorry, when was this? Please provide a citation.

Re: Tell HN: AWS appears to be down again

#642

Post by Gravityloss »

AtlasBarfed wrote: post #537
tyingq wrote: post #264

Earlier quoted context omitted.

"Electrical power systems are designed to be fully redundant so that in the event of a disruption, uninterruptible power supply units can be engaged for certain functions, while generators can provide backup power for the entire facility." https://aws.amazon.com/compliance/data-center/infrastructure... So they have 2 different sources of power coming in. And generators. They do mention the UPS is only for "certain fu…

Has datacenter power redundancy undergone any sort of revolution with grid storage becoming industrial scale? I wonder if a lot of AWS dc design in this area predates the battery grid storage revolution with (what my impression is) a far faster adaptation/switchover time than a generator spin up, and possibly software systems that work to detect and switch over quickly? AWS can claim it will be best of breed, but the…

When deciding data center locations, companies certainly have in mind the quality of the electrical infrastructure in that country or region...

Maybe it could affect people buying services as well.

Re: Tell HN: AWS appears to be down again

#643

Post by phil21 »

laumars wrote: post #476
phil21 wrote: post #401

Earlier quoted context omitted.

You really don't need almost any of this stuff. If you have small on-prem needs just grab a couple fiber links, try for diversity on paths for them (good luck), add some power backup if it fits your needs, and be done. If you are going to the level of the above, you go with co-location in purpose built centers at a wholesale level. The "layer1" is all done to the specs you state and you don't have to worry about it.…

> You really don't need almost any of this stuff. If you have small on-prem needs just grab a couple fiber links, try for diversity on paths for them (good luck), add some power backup if it fits your needs, and be done. Then you’re not running HA and thus the argument about cloud downtime being “worse” than on prem is moot. Obviously if your SLA is basically “we will do our best” then there are all sorts of short cu…

My (late) point is that when people speak about "on-prem" these days, they are not talking about building corporate datacenters on campus.

30 years ago when you talked on-prem that's what this meant. It's now shifted to on-prem meaning your own hardware in massive shared facilities that handle all that "hard stuff" like redundant power and cooling for you.

Bespoke datacenter builds for true-on prem certainly exist, but it's not what that term typically means any longer - at least in my line of business. When I'm selling racks of colo now, my customers are calling that their on-prem facilities.

In fact a large part of my previous business was dismantling true "on-prem" facilities to move to such large shared wholesalers.

Re: Tell HN: AWS appears to be down again

#644

Post by SixDouble5321 »

metadat wrote: post #592
Hippocrates wrote: post #198

Every time a major cloud provider has an outage, Infra people and execs cry foul and say we need to move to . But does anyone really have an objective measure of how clouds stack up reliability-wise? I doubt it, since outages and their effects are nuanced. The other move is that they want to go multi-cloud... But I’ve been involved in enough multi-cloud initiatives to know how much time and effort those soak up, not…

I know the Oracle OCI cloud has a reputation for never going hard-down, but also realize HN seems to loathe Big Red (understandably, to a degree, though OCI is pretty nice IME and _very_ predictable).

I don't think it's unfair. They aren't the worst villain, but they are up there.

Re: Tell HN: AWS appears to be down again

#645

Post by RA_Fisher »

manquer wrote: post #464
RA_Fisher wrote: post #448

Earlier quoted context omitted.

True with two independent servers at 90% each, that’s 0.1^2 = 1% chance both fail— so redundancy can add a lot of reliability.

Only if they are truly independent of each other. You and AWS are using similar chips similar hard disks even with similar failure rates. If you both use same hardware from say batch both can defects and fail at similar times.or you use the same file systems, that say corrupts both your backups. 90% is not a magic number , you need to know AWS supply chains and practices thoroughly and keep yours different enough not…

True. One would want to continually decorrelate services or model the dependencies. Redundancy will help even with some dependency, but you raise an important point.

Re: Tell HN: AWS appears to be down again

#646

Post by fatnoah »

zymhan wrote: post #473
fatnoah wrote: post #410

Earlier quoted context omitted.

My last startup migrated from Verizon Terremark after the healthcare.gov fiasco several years ago. We also suffered from that massive outage and that was the final straw in migrating to AWS. At AWS, we built a few layers of redundant infrastructure with mulit-AZ availability within a region and then global availability across multiple regions. All this was done at roughly half the cost of the traditional hosting, eve…

Indeed, if you only deploy resources in us-east1, or any other single region, you're risking the occasional downtime. I'd wager that will still give you more uptime than a physically-hosted solution for the same cost.

As I discovered many years ago when our infra was only in US-EAST1, failures were also easier to explain since many, many other companies would be offline as well. It made it more of an "Internet problem" than our own company's problem. For whatever reason, customers were far more likely to accept those kinds of outages.
Post reply on HN