Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

231–240 of 646 posts

Re: Tell HN: AWS appears to be down again

#231

Post by dijit »

f6v wrote: post #202
RONROC wrote: post #154

The prevailing wisdom throughout the last couple of years was: “ditch your on-prem infrastructure and migrate to a major cloud provider” And its starting to seem like it could be something like: “ditch your on-prem infrastructure and spin up your own managed cloud” This is probably untenable for larger orgs where convenience gets the blank check treatment, but for smaller operations that can’t realize that value at s…

Self-managed infrastructure doesn’t fail now?

What an absolutely pointless comment.

Everything fails, we can argue the rate. But I would argue that understanding your constraints is better.

if you know that your secret storage system can't survive if a machine goes away: well, you wire redundant paths to the hardware and do memory mirroring and RAID the hell out of the disks. And if it fails you have a standby in place.

But if you use AWS Cognito.

And it goes down.

You're fucked mate.

Re: Tell HN: AWS appears to be down again

#232

Post by mnordhoff »

benedikt wrote: post #189
crescentfresh wrote: post #109

Earlier quoted context omitted.

The affected zone is use1-az4. Whatever that maps to (1a, 1b, 1c) is different per customer.

you can find out which zone is mapped to use1-az4 for your account with awscli: aws ec2 describe-availability-zones | jq -r '.AvailabilityZones[] | select(.ZoneId == "use1-az4") | .ZoneName'

Or if you open the EC2 console (it's up this time!) and scroll down to the bottom.

https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#...:

(Edit: I hope I didn't sound sarcastic. I don't open random console pages and scroll all the way down to check for new features. Some people will have noticed, some won't.)

Re: Tell HN: AWS appears to be down again

#233

Post by uluyol »

ItsBob wrote: post #224

I've built out many 42U racks in DC's in my time and there were a couple of rules that we never skipped: 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power…

Why spend the cost on dual X and Y when you can failover to another cluster?

For big DC workloads, it is usually, though not always, better to take the higher failure rate than add redundancy.

Re: Tell HN: AWS appears to be down again

#234

Post by indigomm »

Hippocrates wrote: post #198

Every time a major cloud provider has an outage, Infra people and execs cry foul and say we need to move to . But does anyone really have an objective measure of how clouds stack up reliability-wise? I doubt it, since outages and their effects are nuanced. The other move is that they want to go multi-cloud... But I’ve been involved in enough multi-cloud initiatives to know how much time and effort those soak up, not…

> I doubt it, since outages and their effects are nuanced.

Your point here deserves highlighting. A failure such as a zone failing is nowadays a relatively simple problem to have. But cloud services do have bugs, internal limits or partial failures that are much more complex. They often require support assistance, which is where the expertise of their staff comes into play. Having a single provider that you know well and trust is better than having multiple providers where you need to keep track of disparate issues.

Re: Tell HN: AWS appears to be down again

#235

Post by Jupe »

trabant00 wrote: post #126
api wrote: post #99

Earlier quoted context omitted.

Nobody ever got fired for using AWS.

True sad fact. I first thought it is a management problem but lately I see it is the tech bros who push for fads in the hopes of staying relevant and not asuming responsability for choices.

(Accidentally down-voted, apologies! I would upvote to fix, but can't... Update: fixed)

Agreed. Arguably, not using an existing cloud service is a red flag on any new hires. AWS being the primary, but experience using GCS or Azure are at least viable skills, even if your business is AWS-based.

But the "fad-based-development" meme is not going away any time soon. The incentives in the business are built around it (really! No one want's to work on a boring old relational database solution any more). In the old days it was 4th generation languages, RUP, XML and Function Point Analysis... today it's functional programming, SDKs, big-three cloud PaaS experience or (shudder) block-chain.

I think back to my much younger self, when I thought that technology was something to be mastered to solve real-world problems, and I laugh. Little did I know the real problem to be solved was to figure out how to solve those same-old business problems but with the technology of the season (Kubernetes, GraphQL or ML).

Re: Tell HN: AWS appears to be down again

#236

Post by Hippocrates »

mijoharas wrote: post #215
Hippocrates wrote: post #198

Every time a major cloud provider has an outage, Infra people and execs cry foul and say we need to move to . But does anyone really have an objective measure of how clouds stack up reliability-wise? I doubt it, since outages and their effects are nuanced. The other move is that they want to go multi-cloud... But I’ve been involved in enough multi-cloud initiatives to know how much time and effort those soak up, not…

I mean from the explanation[0], assuming that is correct (I don't have evidence to suggest it's false) - you don't need to be multi-cloud, and you don't even need to be multi-region. As long as you're spread out over multiple availability zones in a region you should be resilient to this failure. Somewhat surprising to see how many things are failing though, which implies, either that a lot of services aren't able to…

Yeah, my thought is not specific to this scenario. Indeed multi-AZ is a low cost and probably good idea because you often have a shared service management, control plane, and cheap bandwidth between things. Of course, when things fail they often ripple as may be the case here. I don't think clouds have their blast radius perfectly contained and they certainly don't communicate those details well.

One incident I recall was involving our GCP regional storage buckets, which we were using to achieve mutli-region redundancy. One day, both regions went down simultaneously. Google told us that the data was safe but the control plane and API for the service is global. Now I always wonder when I read about MR what that actually means...

Re: Tell HN: AWS appears to be down again

#238

Post by Bluecobra »

ItsBob wrote: post #224

I've built out many 42U racks in DC's in my time and there were a couple of rules that we never skipped: 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power…

> What am I missing?

My guess is that they cheaped out in having redundant PSUs to get you to use multiple availability zones. (More zones = more revenue)

Even a single PSU shouldn’t be an issue if they plugged in an ATS switch though.

Re: Tell HN: AWS appears to be down again

#240

Post by ctvo »

iso1631 wrote: post #211
f6v wrote: post #202

Earlier quoted context omitted.

Self-managed infrastructure doesn’t fail now?

Not at this rate. I remember we had a power outage in 2006, it actually took one of my services off air. Since then of course that has been rectified, and the loss of a building wouldn't impact on any of the critical, essential or important services I provide.

> Not at this rate.

And what rate is this? It gets attention because it impacts more people, but AWS / GCP / Azure uptime is still better than what I've seen for small / mid size businesses trying to manage their own infrastructure.

Post reply on HN