Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

391–400 of 646 posts

Re: Tell HN: AWS appears to be down again

#391

Post by phone8675309 »

tluyben2 wrote: post #361
JCM9 wrote: post #303

AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…

> People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down. Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you…

> Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen.

There are plenty of situations where this might happen if they’re in your room: a lightning strike can cause a surge that causes the disks to fry, a thief might break in and steal your system, your house might burn down, an earthquake could cause your disks to crash, a flood could destroy the machines, and a sinkhole could open up and swallow your house. You may laugh at some of these as being improbable, but I have seen _all_ of these take out systems between my times in Florida (lightning, thief, sinkhole, and flood) and California (earthquake and house fire).

The fix for this is the same fix as being proposed by the parent post - putting physical space between the two systems so if one place become unavailable you still have a backup.

Re: Tell HN: AWS appears to be down again

#392

Post by bob1029 »

2 of our servers are fucked right now. VOIP services down.

Only with AWS and Github do I seem get panicked text messages on my phone first thing in the morning... Our workloads on Azure typically only have faults when everyone is in bed.

Re: Tell HN: AWS appears to be down again

#393

Post by swid »

aeonflux wrote: post #311
kalleth wrote: post #277

Earlier quoted context omitted.

I'd be surprised if they needed backups for a few hours of downtime with (reportedly) complete recovery where no data was corrupted. There are industries where this would be required, and it's possible I guess, but neither of these downtime events were "data loss" events, just availability events for short-ish periods of time that wouldn't - for me - result in activating our DR plans. I must admit that I do always tr…

I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.

One thing you should absolutely not count on, but might be a course of actions for large clients, is to contact support and ask them to restore accidentally / maliciously deleted files.

I would never use this as part of the backup and restore plan; but I was lucky when a bunch of customer files were deleted due to a bug in a release. Something like 100k files were deleted from Google Storage without us having backup. In a panic we contact GCP. We were able to provide a list of all the file names from our logs. In the end, all but 6 files were recovered.

I think it took around 2-3 days to get all the files restored, which was still a big headache and impactful to people.

Re: Tell HN: AWS appears to be down again

#394

Post by acdha »

boudin wrote: post #357
JCM9 wrote: post #303

AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…

Issues are across all us-east 1, not one AZ. Load balancers are not doing well at all. The only way in this case to avoid an outage is to be cross regions or cross cloud which is quite more complex to handle and require more resources to do well. And I hope that nobody is listening your blaming and pointing fingers advice, that's the worst way to solve anything. It's AWS job to ensure that things are reliable, that t…

Some load balancers may be having issues but I have multiple busy workloads showing no issues all morning. One big challenge can be that some people reporting multi-AZ issues are shifting traffic and competing with everyone else, while workloads which were already running in the other AZs were fine. It can be really hard to accurately tell how much the problems you’re seeing generalize to everyone else.

I do agree that the end of this year has been a very bad period for AWS. I wonder whether there’s a connection to the pandemic conditions and the current job market – it feels like a lot of teams are running without much slack capacity, which could lead to both mistakes and longer recovery times.

Re: Tell HN: AWS appears to be down again

#395

Post by davewritescode »

quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

You’re not wrong but there’s ways to do backups properly in AWS and I’m not aware of there ever being an incident where AWS has lost data.

It’s not a bad idea store backups offline but costs might make that an expensive proposition.

Re: Tell HN: AWS appears to be down again

#396

Post by [deleted] »

ItsBob wrote: post #243
uluyol wrote: post #233

Earlier quoted context omitted.

Why spend the cost on dual X and Y when you can failover to another cluster? For big DC workloads, it is usually, though not always, better to take the higher failure rate than add redundancy.

Really? You'd think at Amazon's scale an additional PSU in a 1U custom-built server (I assume they're custom) would be a few tens of $ at most. Actually, now that I type that it makes sense. Scaling a few tens of dollars to a bajillion servers on the off-chance that you get an inbound power failure (quite rare I'd reckon) might cost more than what they'd lose if it does actually fail. So yeah, they're potentially jus…

[deleted]

Re: Tell HN: AWS appears to be down again

#397

Post by [deleted] »

matharmin wrote: post #345
nightpool wrote: post #253

Earlier quoted context omitted.

Amazon is claiming the failure is limited to a single AZ. Are you seeing failures for instances outside of that AZ? If not, how has this rendered "the entire region almost unusable"?

Yes, I've seen issues that affected the entire region. In my specific case, I happened to have an ElastiCache cluster in the affected AZ that became unreachable (my fault for single AZ). But even now, I'm unable to create any new ElastiCache clusters in different AZs (which I wanted to use for manual failover). And there were a lot of errors on the AWS console during the outage. "almost unusable" is maybe exaggeratin…

[deleted]

Re: Tell HN: AWS appears to be down again

#398

Post by dijit »

vel0city wrote: post #387
dijit wrote: post #362

Earlier quoted context omitted.

You don't get fed DC power, you get fed AC power. But, point taken: yes your power feed should be running at Mostly this is outsourced to the datacenter provider; they'll give you a per side rating. (usually 10A or 16A) which also matches the cooling profile of the cabinet.

I mean, in some datacenters they run DC power to each rack. Its definitely more esoteric than having each device run AC but some people do it. However, with their comment DC == Data Center, not Direct Current.

Yeah, I got thrown off by the "per DC supply is 50% of the actual limit"

DC = Datacenter? makes no sense, so my head replaced it with "Power Supply" instead of "DC Supply", second sentence does make sense as being datacenter though.

Re: Tell HN: AWS appears to be down again

#399

Post by lljk_kennedy »

biohax2015 wrote: post #349
dkryptr wrote: post #332

Earlier quoted context omitted.

100% agree. I'm actually surprised AWS hasn't built in a Chaos Monkey into their APIs/console so people can test their resiliency regularly if an AZ goes down. edit: of course, AWS does have this: AWS Fault Injection Simulator

AWS Fault Injection Simulator does this.

Is that what they call us-east-1 nowadays?

Re: Tell HN: AWS appears to be down again

#400

Post by vel0city »

ItsBob wrote: post #243
uluyol wrote: post #233

Earlier quoted context omitted.

Why spend the cost on dual X and Y when you can failover to another cluster? For big DC workloads, it is usually, though not always, better to take the higher failure rate than add redundancy.

Really? You'd think at Amazon's scale an additional PSU in a 1U custom-built server (I assume they're custom) would be a few tens of $ at most. Actually, now that I type that it makes sense. Scaling a few tens of dollars to a bajillion servers on the off-chance that you get an inbound power failure (quite rare I'd reckon) might cost more than what they'd lose if it does actually fail. So yeah, they're potentially jus…

At big cloud provider scale like Amazon, Azure, and Google they probably aren't even running PSUs at each server, they're probably doing DC at the rack these days. No point in having a million little transformers everywhere, far easier maintenance centralizing those and have multiple feeding the bus bars going to each rack.
Post reply on HN