Tell HN: AWS appears to be down again • Hacker News Board

Re: Tell HN: AWS appears to be down again

#251

Post by TrueDuality » Wed, Dec 22, 2021, 2:30 PM UTC

GrumpyNl wrote: post #166

aledalgrande wrote: post #133

If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches wi…

How come they dont have power backups?

According to the SOC certifications they give their customers they do.

Re: Tell HN: AWS appears to be down again

#252

Post by camdenreslink » Wed, Dec 22, 2021, 2:31 PM UTC

Who needs chaos monkey? Just host on AWS for a similar effect.

Re: Tell HN: AWS appears to be down again

#253

Post by nightpool » Wed, Dec 22, 2021, 2:32 PM UTC

codeduck wrote: post #152

aledalgrande wrote: post #133

If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches wi…

another example of a single dc in a single AZ rendering an entire region almost unusable. This has shades of eu-central-1 all over again.

Amazon is claiming the failure is limited to a single AZ. Are you seeing failures for instances outside of that AZ? If not, how has this rendered "the entire region almost unusable"?

Re: Tell HN: AWS appears to be down again

#254

Post by anonu » Wed, Dec 22, 2021, 2:32 PM UTC

Better polish off your BCP docs. People will be asking for them quite a bit more in the new year.

Re: Tell HN: AWS appears to be down again

#255

Post by nextaccountic » Wed, Dec 22, 2021, 2:34 PM UTC

TacticalCoder wrote: post #164

IceWreck wrote: post #50

Honestly my server at home has more uptime than US-East-1

I should blog about this one day but... I have a server at OVH (not affiliated to them) which, at this point, I keep only for fun. It has 3162 days of uptime as I type this. 3 162 days. That's 8 years+ of uptime. Does it have the traffic of Amazon? No. Is it secure? Very likely not: it's running an old Debian version (Debian 7, which came out in, well, 2013). It only has one port opened though, SSH. And with quite a…

> Like that one time I solve LCS35 by computing for about four years on commodity hardware at home.

Awesome! Are you Bernard Fabrot [0]?

[0] https://www.csail.mit.edu/news/programmers-solve-mits-20-yea...

Re: Tell HN: AWS appears to be down again

#256

Post by spmurrayzzz » Wed, Dec 22, 2021, 2:35 PM UTC

omh2 wrote: post #226

mijoharas wrote: post #215

Earlier quoted context omitted.

I mean from the explanation[0], assuming that is correct (I don't have evidence to suggest it's false) - you don't need to be multi-cloud, and you don't even need to be multi-region. As long as you're spread out over multiple availability zones in a region you should be resilient to this failure. Somewhat surprising to see how many things are failing though, which implies, either that a lot of services aren't able to…

AWS doesn't follow their own advice about hosting multi-regional so every time us-east-1 has significant issues pretty much every AZ and region is affected. Specifically large parts of the management API, and IAM service are seemingly centrally hosted in us-east-1. So called Global endpoints are also dependent on us-east-1 and parts of AWS' internal event queues (eg. event bridge triggers) If your infrastructure is s…

Your last comment is really important, I think. I have always petitioned for "passive over active" design in distributed cloud systems. The recent outages, and also ones from the past, demonstrate why.

The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better. Related to that, I have been critical of lambda/FaaS/serverless infrastructure patterns for similar reasons. Always felt like a brittle house of cards to me (N.B. I do still use aws lambda, but keep it constrained to non-critical workloads).

Re: Tell HN: AWS appears to be down again

#257

Post by marcosdumay » Wed, Dec 22, 2021, 2:35 PM UTC

2-718-281-828 wrote: post #203

chkhd wrote: post #176

Earlier quoted context omitted.

"When a fail-safe system fails, it fails by failing to fail-safe." - https://en.wikipedia.org/wiki/Systemantics

is that just playing with words?

You mean to ask if it's a joke? Yes, it's a joke.

Or you ask if it's a lesson about how real systems operate? Because yes, it's a very serious lesson about how real systems operate.

Anyway, you seem out of grasp on system engineering. Your reply downthread isn't applicable (of course fail-safes can fail, anything can fail). If you want to learn more on this area (not everybody wants, and its ok), following that link of system theory books on the wiki may be a good idea. Or maybe start at the root:

https://en.wikipedia.org/wiki/Systems_theory

Notice that there is a huge amount of handwaving in system engineering. I don't think this is good, but I don't think it's avoidable either.

Re: Tell HN: AWS appears to be down again

#258

Post by Jupe » Wed, Dec 22, 2021, 2:36 PM UTC

datavirtue wrote: post #218

trabant00 wrote: post #126

Earlier quoted context omitted.

True sad fact. I first thought it is a management problem but lately I see it is the tech bros who push for fads in the hopes of staying relevant and not asuming responsability for choices.

Omg, this needs to be on a plaque or something. "Let's move our internal app with 50 users to k8s in the cloud." --true story

Amazing. And as long as "technological progress" sufficiently obscures the impact of such ridiculousness the more such projects will continue to occur.

It's a real shame that the collective world of technology does not properly respect the simple solutions that work.

It is almost funny the dichotomy here. Most technological people "admire" the simplicity, elegance and extensibility of the command line. But tell those same people that the best data store for the solution is a relational database and their nose crinkles up.

Re: Tell HN: AWS appears to be down again

#259

Post by spmurrayzzz » Wed, Dec 22, 2021, 2:37 PM UTC

hnarn wrote: post #14

Is there a history of AWS downtimes available somewhere? This makes what, three times in as many months? edit: The question isn't necessarily AWS specific, just any data on amount of downtime per cloud provider on a timeline would be nice.

This is a little more broad, beyond just cloud infra providers, but includes some of the kind of data you're looking for (post-mortems for outage events): https://github.com/danluu/post-mortems

Re: Tell HN: AWS appears to be down again

#260

Post by gtsop » Wed, Dec 22, 2021, 2:38 PM UTC

Question to the sysadmins here: Is it really that outrageous of amazon to have such issues or are people way to spoiled to appreciate the effort that goes into maintaining such a service?

Edit: Not supporting amazon, i generally dislike the company. I just don't understand the extend to which the criticism is justified