Tell HN: AWS appears to be down again • Hacker News Board

Re: Tell HN: AWS appears to be down again

#311

Post by aeonflux » Wed, Dec 22, 2021, 3:05 PM UTC

kalleth wrote: post #277

quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

I'd be surprised if they needed backups for a few hours of downtime with (reportedly) complete recovery where no data was corrupted. There are industries where this would be required, and it's possible I guess, but neither of these downtime events were "data loss" events, just availability events for short-ish periods of time that wouldn't - for me - result in activating our DR plans. I must admit that I do always tr…

I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.

Re: Tell HN: AWS appears to be down again

#312

Post by autosharp » Wed, Dec 22, 2021, 3:05 PM UTC

dijit wrote: post #296

autosharp wrote: post #284

Earlier quoted context omitted.

Also, you can just take two different amazon regions and hope they don't both go down at the same time. For extra safety, and extra work, you could even take Azure as a backup if you're not locked in with AWS.

forgive me repeating myself: AWS Zones are not truly independent of each other. Global services such as route53, Cognito, the default cloud console and Cloudfront are managed out of US-East-1. If us-east-1 is unavailable, as is commonly the case, and you depend on those systems, you are also down. it does not matter if you're in timbuktu-1, you are dead in the water. it is a myth that amazon availability zones are tr…

Of course that depends on what services you use and yes, even then there is some remaining correlation just because it is the same host.

> are not truly independent of each other

Indeed. They are even on the same planet!

> please stop blaming the victim

Excuse me?

Re: Tell HN: AWS appears to be down again

#313

Post by Retric » Wed, Dec 22, 2021, 3:05 PM UTC

lmilcin wrote: post #275

quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted. -- EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on…

It really spends on how reliable you need to be. Don’t forget you get downtime from both AWS and your own issues so even 4 9’s is off the table with pure AWS. If you need to be more reliable than AWS you need to run a hybrid inside and outside of AWS which means most of the advantages of running on AWS goes away.

Re: Tell HN: AWS appears to be down again

#314

Post by NovemberWhiskey » Wed, Dec 22, 2021, 3:06 PM UTC

2-718-281-828 wrote: post #203

chkhd wrote: post #176

Earlier quoted context omitted.

"When a fail-safe system fails, it fails by failing to fail-safe." - https://en.wikipedia.org/wiki/Systemantics

is that just playing with words?

I think it's predicated on a misunderstanding of what "fail-safe" actually means.

For example, in railway signaling, drivers are trained to interpret a signal with no light as the most restrictive aspect (e.g. "danger"). That way, any failure of a bulb in a colored light signal, or a failure of the signal as a whole, results in a safe outcome (albeit that the train might be delayed while the driver calls up the signaler).

Or, in another example from the railways, the air brake system on a train is configured such that a loss of air pressure causes emergency brake activation.

Fail-safe doesn't mean "able to continue operation in the presence of failures"; it means "systematically safe in the presence of failure".

Systems which require "liveness" (e.g. fly-by-wire for a relaxed stability aircraft) need different safety mechanisms because failure of the control law is never safe.

Re: Tell HN: AWS appears to be down again

#315

Post by vinay_ys » Wed, Dec 22, 2021, 3:06 PM UTC

tyingq wrote: post #264

vinay_ys wrote: post #217

Earlier quoted context omitted.

This is quite interesting as they claim their datacenter design does better than Uptime's Tier3+ design requirements which require redundant power supply paths. [ https://aws.amazon.com/compliance/uptimeinstitute/ ]. I really hope they publish a thorough RCA for this incident.

"Electrical power systems are designed to be fully redundant so that in the event of a disruption, uninterruptible power supply units can be engaged for certain functions, while generators can provide backup power for the entire facility." https://aws.amazon.com/compliance/data-center/infrastructure... So they have 2 different sources of power coming in. And generators. They do mention the UPS is only for "certain fu…

Usually when someone claims T3+ they mean they have UPS clusters in 3+1 (or such) configuration and two different such UPS clusters power two power-strips in a rack. Then, would also have incoming grid power supply from two different HV sub-stations with non-intersecting cable paths. They would also have diesel power generators in 3+1 or 5+2 configurations with automatic startup time in seconds. The UPS's power storage (chemical or potential energy based devices) can hold enough energy to handle full load for several minutes. If these are design and maintained correctly, even while concurrent scheduled maintenance is ongoing, an unexpected component failure should not cause catastrophic outage. At each layer (grid incomers, generator incomers, UPS power incomers) there are switches to switch over whenever there's a need (maintenance or failure).

If they claim tier4, then they basically have everything in n+n configuration.

Re: Tell HN: AWS appears to be down again

#316

Post by Spooky23 » Wed, Dec 22, 2021, 3:07 PM UTC

mbesto wrote: post #228

iso1631 wrote: post #211

Earlier quoted context omitted.

Not at this rate. I remember we had a power outage in 2006, it actually took one of my services off air. Since then of course that has been rectified, and the loss of a building wouldn't impact on any of the critical, essential or important services I provide.

> Not at this rate. Source? Has there ever been an industry wide survey that compares availability from "insert average colo/data center operations" with the cloud ones? And I'm not talking about "we have 12 SREs who are based in Cupertino and are all paid top dollar to support a colo"...I'm talking average .

Running a multi-tenant datacenter or hyper scale cloud datacenter is a different business than running a datacenter. The myth of the cost of running facilities on HN is insane - it’s like saying you can’t drive a car unless you hire a formula 1 driver.

I worked through the ranks at a large enterprise that ran a “big” datacenter for a decade. The facilities team was about 6 people, average salary around $90k. I can only remember one power interruption affecting more than a rack, caused by a failure during a maintenance event that requires a shutdown for safety reasons. The rest is like any other industrial facility - you have service contracts for the equipment, etc and maintain things.

There’s a cost/capability curve that you need to plan around for these matters. You need to make business and engineering decisions based on your actual circumstances. If the answer is automatically “AWS “, you’re making a decision to burn dollars for convenience.

Re: Tell HN: AWS appears to be down again

#317

Post by lordnacho » Wed, Dec 22, 2021, 3:07 PM UTC

ItsBob wrote: post #224

I've built out many 42U racks in DC's in my time and there were a couple of rules that we never skipped: 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power…

What about a UPS/battery thingy? That's saved me a few times, though it normally just gives enough time for a short outage. Is it uncommon in cloud infra?

Re: Tell HN: AWS appears to be down again

#318

Post by exabrial » Wed, Dec 22, 2021, 3:07 PM UTC

Stat That.

Re: Tell HN: AWS appears to be down again

#319

Post by notyourday » Wed, Dec 22, 2021, 3:07 PM UTC

ItsBob wrote: post #249

notyourday wrote: post #241

Earlier quoted context omitted.

> I've only had a DC fail once when the engineer was performing work on the power circuitry for the DC and thought he was taking down one, but was in fact the wrong one and took both power circuits down at the same time. This is all local scale. Your setup would not survive a data center scale power outage. At scale power outages are datacenter scale. Data centers lose supply lines. They lose transformers. Sometimes…

Yes but if you have reliable power from two different sources then the biggest risk (I'd imagine) is the failover circuitry! Something that should be tested tbh. Also, there are banks of batteries and generators in between the power company cables and the kit: did they not kick-in? Again, this is all pure speculation: I have absolutely no idea of the exact failure, nor how their infrastructure is held together - this…

> Yes but if you have reliable power from two different sources then the biggest risk (I'd imagine) is the failover circuitry! Something that should be tested tbh.

That's ATS. It is not really advisable to test their under load performance because the failure of an ATS would be catastrophic. ATS typically would be tested at the installation and after that their parameters would be monitored.

Replacing a functional in line ATS would be a 9-12 months long project.

> Also, there are banks of batteries and generators in between the power company cables and the kit: did they not kick-in?

At high energy you are pretty much always going to use an ATS.

Re: Tell HN: AWS appears to be down again

#320

Post by reilly3000 » Wed, Dec 22, 2021, 3:08 PM UTC

londons_explore wrote: post #276

nightpool wrote: post #253

Earlier quoted context omitted.

Amazon is claiming the failure is limited to a single AZ. Are you seeing failures for instances outside of that AZ? If not, how has this rendered "the entire region almost unusable"?

A lot of people will automatically fail over jobs to other AZ's. That often involves spinning up lots more EC2 instances and moving PB's of data. The end result is all capacity on other AZ's gets used up, and networks get full to capacity, and even if those other zones are technically working, practically they aren't really usable.

While there may be more machines provisioned, many orgs run active setups for failover so they aren’t as affected. In terms of data transfer, it should already be there. Where would it come from? Certainly not the dead AZ.