Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

531–540 of 646 posts

Re: Tell HN: AWS appears to be down again

#531

Post by dijit »

f6v wrote: post #487
dijit wrote: post #231

Earlier quoted context omitted.

What an absolutely pointless comment. Everything fails, we can argue the rate. But I would argue that understanding your constraints is better. if you know that your secret storage system can't survive if a machine goes away: well, you wire redundant paths to the hardware and do memory mirroring and RAID the hell out of the disks. And if it fails you have a standby in place. But if you use AWS Cognito. And it goes do…

It’s pointless to discuss how crappy cloud is whenever AWS goes down. Most of the businesses relying by the automatic RDS backups or EC2 auto scaling just don’t have time to think about all the underlying tech. I mean, I don’t manually allocate memory for variables anymore either. Do I get screwed when there’s a memory leak? Yes. What do I do about it? Move on.

Then don’t host anything, don’t do software and don’t pretend to be “the future”.

Re: Tell HN: AWS appears to be down again

#532

Post by wizwit999 »

matharmin wrote: post #345
nightpool wrote: post #253

Earlier quoted context omitted.

Amazon is claiming the failure is limited to a single AZ. Are you seeing failures for instances outside of that AZ? If not, how has this rendered "the entire region almost unusable"?

Yes, I've seen issues that affected the entire region. In my specific case, I happened to have an ElastiCache cluster in the affected AZ that became unreachable (my fault for single AZ). But even now, I'm unable to create any new ElastiCache clusters in different AZs (which I wanted to use for manual failover). And there were a lot of errors on the AWS console during the outage. "almost unusable" is maybe exaggeratin…

That seems acceptable. The Data plane failure is contained to an AZ. Control plane is often not.

Re: Tell HN: AWS appears to be down again

#533

Post by fulafel »

SCdF wrote: post #305
aledalgrande wrote: post #133

If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches wi…

So dumb question from someone who hasn't maintained large public infrastructure: Isn't the whole point of availability zones is that you deploy to more than one and support failing over if one fails? IE why are we (consumers) hearing about this or being obviously impacted (eg Epic Games Store is very broken right now)? Is my assessment wrong, or are all these apps that are failing built wrong? Or something in between…

IME people rarely test and drill for the failovers, it's just a checkbox in a high level plan. Maybe they have a todo item for it somewhere but it never seems very important as AZ failures are usually quite rare. After ignoring the issue for a while it starts to seem risky to test for it, you might get an outage due to bugs it's likely to uncover.

Re: Tell HN: AWS appears to be down again

#535

Post by sprite »

SCdF wrote: post #305
aledalgrande wrote: post #133

If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches wi…

So dumb question from someone who hasn't maintained large public infrastructure: Isn't the whole point of availability zones is that you deploy to more than one and support failing over if one fails? IE why are we (consumers) hearing about this or being obviously impacted (eg Epic Games Store is very broken right now)? Is my assessment wrong, or are all these apps that are failing built wrong? Or something in between…

I thought I was Multi AZ but something failed. I am mostly running EC2 + RDS both with 2 availability zones. I will have to dig into the problem but I think the issue is that my setup for RDS is one writer instance and one reader instance, each in a different AZ. However I guess there was nothing for it to fail over to since my other instance was the writer instance, so I guess I need to keep a 3rd instance available preferably in a 3rd AZ?

Re: Tell HN: AWS appears to be down again

#536

Post by belfalas »

notyourday wrote: post #319
ItsBob wrote: post #249

Earlier quoted context omitted.

Yes but if you have reliable power from two different sources then the biggest risk (I'd imagine) is the failover circuitry! Something that should be tested tbh. Also, there are banks of batteries and generators in between the power company cables and the kit: did they not kick-in? Again, this is all pure speculation: I have absolutely no idea of the exact failure, nor how their infrastructure is held together - this…

> Yes but if you have reliable power from two different sources then the biggest risk (I'd imagine) is the failover circuitry! Something that should be tested tbh. That's ATS. It is not really advisable to test their under load performance because the failure of an ATS would be catastrophic. ATS typically would be tested at the installation and after that their parameters would be monitored. Replacing a functional in…

> the failure of an ATS would be catastrophic

Because that would mean no power at all to the DC and no way to get it back? (I am completely ignorant on this topic)

Re: Tell HN: AWS appears to be down again

#537

Post by AtlasBarfed »

tyingq wrote: post #264
vinay_ys wrote: post #217

Earlier quoted context omitted.

This is quite interesting as they claim their datacenter design does better than Uptime's Tier3+ design requirements which require redundant power supply paths. [ https://aws.amazon.com/compliance/uptimeinstitute/ ]. I really hope they publish a thorough RCA for this incident.

"Electrical power systems are designed to be fully redundant so that in the event of a disruption, uninterruptible power supply units can be engaged for certain functions, while generators can provide backup power for the entire facility." https://aws.amazon.com/compliance/data-center/infrastructure... So they have 2 different sources of power coming in. And generators. They do mention the UPS is only for "certain fu…

Has datacenter power redundancy undergone any sort of revolution with grid storage becoming industrial scale?

I wonder if a lot of AWS dc design in this area predates the battery grid storage revolution with (what my impression is) a far faster adaptation/switchover time than a generator spin up, and possibly software systems that work to detect and switch over quickly?

AWS can claim it will be best of breed, but they aren't going to throw out a DC power redundancy investment (or threaten downtime) that they can't wring more ROI on.

Re: Tell HN: AWS appears to be down again

#539

Post by luhn »

peeters wrote: post #344
SCdF wrote: post #305

Earlier quoted context omitted.

So dumb question from someone who hasn't maintained large public infrastructure: Isn't the whole point of availability zones is that you deploy to more than one and support failing over if one fails? IE why are we (consumers) hearing about this or being obviously impacted (eg Epic Games Store is very broken right now)? Is my assessment wrong, or are all these apps that are failing built wrong? Or something in between…

As I understand it for something like SQS, Lambda etc, AWS should automatically tolerate an AZ going down. They're responsible for making the service highly available. For something like EC2 though, where a customer is just running a node on AWS, there's no automatic failover. It's a lot more complicated to replicate a running, stateful virtual machine and have it seamlessly failover to a different host. So typically…

It sounds like EC2 API is having a brownout due to this, so a lot of people can't failover to a new AZ.

Re: Tell HN: AWS appears to be down again

#540

Post by BenjiWiebe »

dylan604 wrote: post #407
tyingq wrote: post #264

Earlier quoted context omitted.

"Electrical power systems are designed to be fully redundant so that in the event of a disruption, uninterruptible power supply units can be engaged for certain functions, while generators can provide backup power for the entire facility." https://aws.amazon.com/compliance/data-center/infrastructure... So they have 2 different sources of power coming in. And generators. They do mention the UPS is only for "certain fu…

The generators should be powering up as soon as one of the 2 different sources goes down. It takes generators a few minutes to power up and get "warmed up". If they don't start this process until both mains sources are down, then oops, there's power outage. I used to work next door to a "major" cable TV station's broadcast location. They had multiple generators on-site, and one of them was running 24/7 (they rotated…

Are you sure about the few minutes part? The standby generators I've seen take seconds to go from off to full load. We have an 80kw model, but I've also seen videos of load tests of much larger generators and they also take only seconds to go to full load.
Post reply on HN