Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

551–560 of 646 posts

Re: Tell HN: AWS appears to be down again

#551

Post by KronisLV »

madeofpalk wrote: post #428
ClumsyPilot wrote: post #271

Now that everyone and their dog is on AWS, it is not just 'a website stops working', half the world, from telephones to security doors and Iot equipment, stops working? I am not sure if the movement the cloud has reduced amount of failures, but it definitely has made these failures more catastrophic. Our profession is busy makin the world less reliable and more fragile, we will have our reconning just like the shippi…

all I've noticed is slack was a bit unreliable for a little bit, but i just carried on and otherwise ignored it. my world did not stop working.

Same, maybe that was a related issue.

Today, on Slack i could not edit messages, could not edit statuses and could not post attachments. Pretty annoying!

Re: Tell HN: AWS appears to be down again

#552

Post by philsnow »

xyst wrote: post #308
aledalgrande wrote: post #133

If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches wi…

This region in general is a clusterfuck. If companies by now do not have a disaster recovery and resiliency strategy in place, you are just shooting yourself in the foot.

In today's world of stitching together dozens of services, who each probably do the same thing, how is one to avoid a dependency on us-east-1? Add yet another bullet to the vendor questionnaire (ugh) about whether they are singly-homed / have a failover plan?

It's turtles all the way down, and underneath all the turtles is us-east-1.

Re: Tell HN: AWS appears to be down again

#553

Post by [deleted] »

laumars wrote: post #497
bcrosby95 wrote: post #490

Earlier quoted context omitted.

Most people using AWS aren't using multi-region, as evidenced by the wide array of problems on the internet when a region goes down. I would also argue many aren't even using multiple availability zones, as evidenced by the wide array of problems on the internet when a single AZ goes down. I think you're vastly over-estimating how most companies are using AWS, and are substituting your own requirements for theirs. Wh…

I fully believe some people are doing AWS wrong. But you cannot compare the worst offenders in AWS against the best defenders of deploying on prem - it’s just not a fair like for like comparison comparing the worst against the best. Hence why I compare doing HA in AWS correctly vs doing HA on prem correctly.

[deleted]

Re: Tell HN: AWS appears to be down again

#554

Post by tyingq »

AtlasBarfed wrote: post #537
tyingq wrote: post #264

Earlier quoted context omitted.

"Electrical power systems are designed to be fully redundant so that in the event of a disruption, uninterruptible power supply units can be engaged for certain functions, while generators can provide backup power for the entire facility." https://aws.amazon.com/compliance/data-center/infrastructure... So they have 2 different sources of power coming in. And generators. They do mention the UPS is only for "certain fu…

Has datacenter power redundancy undergone any sort of revolution with grid storage becoming industrial scale? I wonder if a lot of AWS dc design in this area predates the battery grid storage revolution with (what my impression is) a far faster adaptation/switchover time than a generator spin up, and possibly software systems that work to detect and switch over quickly? AWS can claim it will be best of breed, but the…

I'd be surprised. Data centers eat a lot of energy, and it's hard to beat the energy density of diesel (120 MJ/kg vs ~1 for batteries) and the ability to have nearby tanks or scheduled trucks.

Tesla apparently did some early pilot stuff: https://www.datacenterdynamics.com/en/analysis/teslas-powerp...

Re: Tell HN: AWS appears to be down again

#555

Post by matharmin »

zymhan wrote: post #478
matharmin wrote: post #338

Earlier quoted context omitted.

AWS is under-reporting the severity of the issue though. The primary outage may be in a single AZ, but there are parts of the AWS stack that affected all AZs in us-east-1, and potentially other regions as well. For example, even now I'm unable to create a new ElastiCache cluster in different AZs of us-east-1.

> I'm unable to create a new ElastiCache cluster in different AZs of us-east-1 Isn't that because Elasticache will distribute the cluster across AZs automatically? https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/...

In this case, this was specifically with a single-AZ setup, using an AZ that was supposed to be unaffected.

Re: Tell HN: AWS appears to be down again

#556

Post by Spivak »

paulryanrogers wrote: post #177
RONROC wrote: post #154

The prevailing wisdom throughout the last couple of years was: “ditch your on-prem infrastructure and migrate to a major cloud provider” And its starting to seem like it could be something like: “ditch your on-prem infrastructure and spin up your own managed cloud” This is probably untenable for larger orgs where convenience gets the blank check treatment, but for smaller operations that can’t realize that value at s…

Spread the risk? Smaller on prem and cloud / rented bare metal?

Nah, it's actually better to concentrate the risk in this case.

If your app depends on a few 3rd party services -- SendGrid, Twilio, Okta and they're all hosted on different infra then congrats! You're gonna have issues when any one of them are down, yayyy.

Also the marketing benefit can't be downplayed. If your postmortem is "AWS was having issues" then your execs and customers just accept that as the cost of doing business because there's a built-in assumption that AWS, Azure, GCP are world class and any in-house team couldn't do it better.

Re: Tell HN: AWS appears to be down again

#558

Post by Retric »

nostrebored wrote: post #363
Retric wrote: post #313

Earlier quoted context omitted.

It really spends on how reliable you need to be. Don’t forget you get downtime from both AWS and your own issues so even 4 9’s is off the table with pure AWS. If you need to be more reliable than AWS you need to run a hybrid inside and outside of AWS which means most of the advantages of running on AWS goes away.

Very untrue. Many businesses with 4 9 SLAs are all in on AWS. It requires active/active setups though!

Many business claim 4 9 SLAs on AWS, but that doesn’t mean they actually provide it. It’s simply a question of what the penalties of failing to reach their SLA is.
Post reply on HN