Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

541–550 of 646 posts

Re: Tell HN: AWS appears to be down again

#541

Post by dehrmann »

ClumsyPilot wrote: post #271

Now that everyone and their dog is on AWS, it is not just 'a website stops working', half the world, from telephones to security doors and Iot equipment, stops working? I am not sure if the movement the cloud has reduced amount of failures, but it definitely has made these failures more catastrophic. Our profession is busy makin the world less reliable and more fragile, we will have our reconning just like the shippi…

It's more like it's making downtimes correlated rather than random. For everything other than urgent communication, I'm not sure if this is a big deal.

Re: Tell HN: AWS appears to be down again

#542

Post by TameAntelope »

JCM9 wrote: post #303

AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…

Here's a secret that's now saved me from three outages this month:

Be in multiple AZs, and even multiple regions but if you're going to be in only one AZ or one region, make it us-east-2.

Re: Tell HN: AWS appears to be down again

#543

Post by [deleted] »

ItsBob wrote: post #224

I've built out many 42U racks in DC's in my time and there were a couple of rules that we never skipped: 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power…

[deleted]

Re: Tell HN: AWS appears to be down again

#544

Post by TameAntelope »

RONROC wrote: post #154

The prevailing wisdom throughout the last couple of years was: “ditch your on-prem infrastructure and migrate to a major cloud provider” And its starting to seem like it could be something like: “ditch your on-prem infrastructure and spin up your own managed cloud” This is probably untenable for larger orgs where convenience gets the blank check treatment, but for smaller operations that can’t realize that value at s…

I don't think it's reasonable to be spooked by these outages, and to think your resolution would be to leave AWS entirely.

A much faster and more effective solution that doesn't have you trading cloud problems with on-prem problems (the power outage still happens, except now it's your team that has to handle it) would be to update your services to run in multiple AZs and multiple regions.

Get out of AWS is you want, but don't get out of AWS because of outages. You should be able to mitigate this relatively easily.

Re: Tell HN: AWS appears to be down again

#545

Post by reaperducer »

BenjiWiebe wrote: post #540
dylan604 wrote: post #407

Earlier quoted context omitted.

The generators should be powering up as soon as one of the 2 different sources goes down. It takes generators a few minutes to power up and get "warmed up". If they don't start this process until both mains sources are down, then oops, there's power outage. I used to work next door to a "major" cable TV station's broadcast location. They had multiple generators on-site, and one of them was running 24/7 (they rotated…

Are you sure about the few minutes part? The standby generators I've seen take seconds to go from off to full load. We have an 80kw model, but I've also seen videos of load tests of much larger generators and they also take only seconds to go to full load.

It might depend on when the backup system was built. No company updates their system every year.

A few minutes seems correct for one place I worked.

This was back in the 90's, before UPS technology got really interesting. Our system was two large rooms with racks and racks and racks of car batteries wired together. When the power went out, the batteries took over until the diesel generator could come online.

I saw it work during several hurricanes and other flood events.

I always found the idea of running an entire building off of car batteries amusing. The engineers didn't share my mirth.

Re: Tell HN: AWS appears to be down again

#546

Post by acdha »

boudin wrote: post #449
acdha wrote: post #394

Earlier quoted context omitted.

Some load balancers may be having issues but I have multiple busy workloads showing no issues all morning. One big challenge can be that some people reporting multi-AZ issues are shifting traffic and competing with everyone else, while workloads which were already running in the other AZs were fine. It can be really hard to accurately tell how much the problems you’re seeing generalize to everyone else. I do agree th…

I hope AWS will provide some explanation about those issues and what actions they will take to prevent those in the future On our side we saw some EC2 VM totally disconnected from the network in 3 AZs.

Yeah, definitely needs good visibility. They’re asking customers to trust them to a large degree.

Re: Tell HN: AWS appears to be down again

#547

Post by greiskul »

phone8675309 wrote: post #391
tluyben2 wrote: post #361

Earlier quoted context omitted.

> People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down. Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you…

> Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen. There are plenty of situations where this might happen if they’re in your room: a lightning strike can cause a surge that causes the disks to fry, a thief might break in and steal your system, your house might burn down, an earthquake could cause your disk…

I have had a job where my small, internal tool, for debugging purposes, had to be deployed to a minimum of 3 datacenters. I had 2 of them in the US and one in Europe, and was asked to move one of the US ones to a datacenter that was in another coast, cause who knows, maybe an earthquake will knock off all of the US west coast. That is the paranoia level necessary to achieve crazy high uptime.

Re: Tell HN: AWS appears to be down again

#548

Post by dehrmann »

schnebbau wrote: post #25

So, how many execs are going to push to move to self-managed hosting in the new year? Packaging a way to migrate off AWS could be a unicorn idea.

Depends on how many customers are ready to move to a different vendor. I suspect most customers are forgiving because either they were also down or half the services they use were down. You don't get fired for hosting in AWS.

Re: Tell HN: AWS appears to be down again

#549

Post by azernik »

junon wrote: post #122
azernik wrote: post #98

Earlier quoted context omitted.

That's more of a trademark issue, and would require a reasonable consumer to be likely to be deceived. Which they're not.

No, it's not a trademark issue. They copied the work verbatim (including code, which is not covered by trademark law, but by copyright law), modified it, and then put the original copyright notice in the legalese. This is copyright infringement. And consumers are clearly deceived - hence why my original comment asking about it was written and has several upvotes.

The direct copyright is covered by the satire exceptions. If you want to argue that copying the legalese is different, it's going to be on those confusion grounds, which IIUC aren't a copyright concern.

Re: Tell HN: AWS appears to be down again

#550

Post by Spivak »

dijit wrote: post #531
f6v wrote: post #487

Earlier quoted context omitted.

It’s pointless to discuss how crappy cloud is whenever AWS goes down. Most of the businesses relying by the automatic RDS backups or EC2 auto scaling just don’t have time to think about all the underlying tech. I mean, I don’t manually allocate memory for variables anymore either. Do I get screwed when there’s a memory leak? Yes. What do I do about it? Move on.

Then don’t host anything, don’t do software and don’t pretend to be “the future”.

This makes no sense. This has nothing to do with the tech and more to do with every team's natural push and pull with build over buy. It's completely pointless to respond to someone who didn't get their DoorDash order with "see this is why you should just make food at home." It completely ignores the reason someone chose to order takeout in the first place.
Post reply on HN