Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-2 outage

news.ycombinator.com

Post reply on HN
Print view View original

221–230 of 254 posts

Re: AWS us-east-2 outage

#221

Post by leesalminen »

happymellon wrote: post #195
blamarvt wrote: post #165

Earlier quoted context omitted.

If your application and infra can magically utilize multiple zones with “a couple lines”… then I would say you are miles ahead of just about every other web company.

> you are miles ahead of just about every other web company. I'm curious who these web companies are. Use something like Lambda and you get multi-az for free. https://docs.aws.amazon.com/lambda/latest/dg/security-resili... Dynamo is another service that wouldn't be impacted as it is multi-az. Getting postgres RDS multi-region would require the extra couple of lines in your CDK, but is fairly straightforward.

Today, a SaaS I’m familiar with that runs ~10 Aurora clusters in us-east-2 with 2-3 nodes each (1 writer, 1-2 readers) in different AZs had prolonged issues.

At least 1 cluster had a node on “affected” hardware (per AWS). Aurora failed to failover properly and the cluster ended up in a weird error state, requiring intervention from AWS. Could not write to the db at all. This took several hours to resolve.

All that to say that it’s never straightforward. In today’s event, it was pure luck of the draw as to whether a multi-AZ Aurora cluster was going to have >60 seconds of pain.

That SaaS has been running Aurora for years and has never experienced anything similar. I was very surprised when I heard the cluster was in a non-customer-fixable state and required manual intervention. I’ve shilled Aurora hard. Now I’m unsure.

Thank goodness they had an enterprise support deal or who knows if they’d still have issues now.

Re: AWS us-east-2 outage

#222

Post by scubbo »

dmalvarado wrote: post #126

I dunno how else to put it. Having EVERYTHING on AWS is a national security threat. This isn't good, and someone who can do something about it needs to.

Having _everything_ on a single AZ of AWS is, indeed, a problem.

Having everything well-architected on AWS is...well, it's a problem for reasons of monopoly and cost, but it's not a problem for availability.

Re: AWS us-east-2 outage

#223

Post by throwbigdata »

saimiam wrote: post #190
willmadden wrote: post #181

Earlier quoted context omitted.

Salaries are a cost.

To an investor, a salary is a temporary cost ie, you pay the salary, get the TF scripts made, fire the employee while a checkbox driven, managed resiliency is going to cost you forever with no hope of ever eliminating that cost. At least that’s what was recently told to me by my manager to explai why my employer prefers to hire people to self manage the AWS infra.

Wait. Are you saying that while AWS maintains multiple AZs they can’t maintain reliability on the failover systems between them?

Re: AWS us-east-2 outage

#224

Post by throwbigdata »

throwbigdata wrote: post #209
jedberg wrote: post #162

Earlier quoted context omitted.

A single AZ may have neighboring datacenters, but they are very strict on having datacenters for different AZs be at least 100km apart and on different flood plains and power grids.

100km? Oh really?

https://docs.aws.amazon.com/sap/latest/general/arch-guide-ar...

Each Availability Zone can be multiple data centers.At full scale, it can contain hundreds of thousands of servers. They are fully isolated partitions of the AWS global infrastructure. With its own powerful infrastructure, an Availability Zone is physically separated from any other zones. There is a distance of several kilometers, although all are within 100 km (60 miles of each other).

Re: AWS us-east-2 outage

#225

Post by tetha »

justapassenger wrote: post #157
dangero wrote: post #101

Earlier quoted context omitted.

For most businesses a little down time here and there is a calculated risk versus more complex infrastructure. You can’t assume all the cloud architects are idiots — they have to report their task list and cost of infrastructure to someone who can give feedback on various options based on comparative resource requirements and risks. Zone downtime still falls under an AWS SLA so you know about how much downtime to acc…

This. People working in IT naturally think keeping IT systems up 100% time is most important. And depending on the business it often is, but it all costs money. Running a business is about managing costs and risks. - Is it worth to spend 20% more on IT to keep our site up 99.99% vs 99%? - Is it worth to have 3 suppliers for every part that our business depends, with each of them being contracted to be able to supply…

I've had some interesting discussions about this with a bunch of representatives of our larger B2B customers about this. Interestingly enough, to them, a controlled downtime of 2-4 hours with an almost guaranteed success is preferable, compared to a more complex, probably working zero-downtime effort that might leave the system in a messed up - or not messed up - state.

To them it's much easier to communicate "Hey, our customer service is going to be degraded on the second saturday in october, call on monday" to their customers 1-2 month in advance, prepare to have the critical information without our system, and have agents tell people just that.

This has really started to change my thoughts of how to approach, e.g. a major postgres update. In our case, it's probably the better way to just take a backup, shutdown everything, do an offline upgrade and rebuild & restore if things unexpectedly go wrong. We can totally test the happy case to death, and if the happy case works, we're done in 2 hours for the largest systems with minimal risk. 4 hours if we have to recover from nothing, also tested.

And you know, at that point, is it really economical to spend weeks to plan and weeks to test a zero downtime upgrade that's hard to test, because of load on the cluster?

Re: AWS us-east-2 outage

#226

Post by chronid »

bigfatfrock wrote: post #32

Suppose it's time to setup multi-az and pay to insure against AWS' own failures. I don't know why I previously thought their EC2 uptime claims were sufficient. Lesson learned.

Multi-AZ is a requirement on production level loads if you cannot sustain prolonged downtime.

Datacenters do end up completely dying now and then, you really want to have a good strategy in that case. Or not, if that's not required.

Re: AWS us-east-2 outage

#227

Post by laluser »

jedberg wrote: post #162
Tehnix wrote: post #120

Earlier quoted context omitted.

> each AZ is considered a separate datacenter but in real life each AZ is multiple datacenters that are really close to each other For AWS specifically, I’m fairly certain they maintain a minimum distance and are much more strict on requirements to be on different grids etc than other Cloud providers. A few years ago they were calling out Azure and Google Cloud on exactly what you describe (having data centers essent…

A single AZ may have neighboring datacenters, but they are very strict on having datacenters for different AZs be at least 100km apart and on different flood plains and power grids.

This should be at most 100km. Range is in 60km-100km range typically.

Re: AWS us-east-2 outage

#228

Post by tomxor »

apengwin wrote: post #156
sergiotapia wrote: post #130

Earlier quoted context omitted.

I'm moving to us-weast-1

what kind of compass are ya reading lad?

I presume they are trying to express an extra cardinal dimension perpendicular to the plane. Deep underground in Bezos new evil lair perhaps?

Re: AWS us-east-2 outage

#229

Post by sergiotapia »

tomxor wrote: post #228
apengwin wrote: post #156

Earlier quoted context omitted.

what kind of compass are ya reading lad?

I presume they are trying to express an extra cardinal dimension perpendicular to the plane. Deep underground in Bezos new evil lair perhaps?

https://www.youtube.com/watch?v=RuJNUXT2a9U

Re: AWS us-east-2 outage

#230

Post by happymellon »

the_fury wrote: post #204
happymellon wrote: post #196

Earlier quoted context omitted.

If you had "done the math" then you would have gone serverless and gained multi-az for free, as it is almost always the cheapest option.

This has quickly grown to more than adding in a couple of lines! Now I need to architect my legacy app so that I can deploy into lambdas, then I can get resiliency I don't really need! Not all systems require high availability. Some systems are A-OK with downtime. Sometimes, I'm perfectly fine with eventual consistency. You really do have to look at the use-cases and requirements before making sweeping staements.

I thought we were talking about cloud architects making poor decisions when designing solutions.

Where did legacy apps come from?

> Some systems are A-OK with downtime.

And those ones would not have cared about this outage. Your point isn't that clear.

Post reply on HN