Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-2 outage

news.ycombinator.com

Post reply on HN
Print view View original

201–210 of 254 posts

Re: AWS us-east-2 outage

#201

Post by creeble »

arecurrence wrote: post #70
acwan93 wrote: post #65

Earlier quoted context omitted.

In all seriousness, we've been deploying everything on us-west-2, and it seems to have dodged most of the outages recently. Is there something special about that data center?

Classically, us-east-1 received most of the hate given its immense size (it used to be several times larger than any other) and status as the first large aws data center. It also seemed to launch new aws features first but that may have been my imagination. If true, I'm sure always running the latest builds was not great for stability. us-west-2 has had outages as well but it is less common, even rare. I've been push…

Doesn't AWS IoT run only in us-east-1?

And I think Alexa skills, if anybody cares about those.

Re: AWS us-east-2 outage

#203

Post by dexterdog »

fishnchips wrote: post #55
jedberg wrote: post #44

Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2." Now I guess we have to move to us-west-2. :) Update: looks like it's only one zone anyway, so my statement still stands!

Stay in us-east-1, they provide Chaos Monkey for free. It's a feature.

I always say stay in use1 because almost everybody is there and when it's suffering any kind of outage so much of the internet is affected that it's no big deal that you're a part of the outage. People just go outside and get some air knowing it will be back up in a few hours, usually right around the time the AWS status page acknowledges that there is an issue.

Re: AWS us-east-2 outage

#204

Post by the_fury »

happymellon wrote: post #196
the_fury wrote: post #175

Earlier quoted context omitted.

Or how about "I'm fully aware, I've done the math taking into account both cost and complexity of implementation and cost of downtime, and I'm probably making fantastic calls based on my actual needs."

If you had "done the math" then you would have gone serverless and gained multi-az for free, as it is almost always the cheapest option.

This has quickly grown to more than adding in a couple of lines! Now I need to architect my legacy app so that I can deploy into lambdas, then I can get resiliency I don't really need!

Not all systems require high availability. Some systems are A-OK with downtime. Sometimes, I'm perfectly fine with eventual consistency. You really do have to look at the use-cases and requirements before making sweeping staements.

Re: AWS us-east-2 outage

#205

Post by twistedpair »

blamarvt wrote: post #165
happymellon wrote: post #154

Earlier quoted context omitted.

Considering almost all of the services are multi-zone, it's not hard to add in a couple of lines to make them resilient against this. People are just unaware, and probably making bad calls in the name of being "portable".

If your application and infra can magically utilize multiple zones with “a couple lines”… then I would say you are miles ahead of just about every other web company.

It's that easy for a lot of managed services.

Want GKE to run multi-zone, or Spanner to run multi-region, just check a box (and insert coin).

Re: AWS us-east-2 outage

#206

Post by xchaotic »

aledalgrande wrote: post #185
Sebb767 wrote: post #177

Earlier quoted context omitted.

If you can get the data out of the downed AZ, don't have state you need to transfer and are not shot in the foot once the primary replica comes online again. I've rarely deployed an app where it was as easy as just to change a region variable.

Yeah the data stores are the ones that I would always keep multi AZ no matter what. Everything else is stateless and can be moved quickly.

Write an article on that because you make it sound simple. Or better yet, start a company that configures this for companies.

Re: AWS us-east-2 outage

#207

Post by fnordpiglet »

Sebb767 wrote: post #171
justapassenger wrote: post #157

Earlier quoted context omitted.

This. People working in IT naturally think keeping IT systems up 100% time is most important. And depending on the business it often is, but it all costs money. Running a business is about managing costs and risks. - Is it worth to spend 20% more on IT to keep our site up 99.99% vs 99%? - Is it worth to have 3 suppliers for every part that our business depends, with each of them being contracted to be able to supply…

At least in my experience, AWS downtime also only accounts for a minor share of the total downtime; the major source are crashes and bugs in the application you're actually trying to host. Being completely HA and independent of AZ crashes/bug is extremely hard and time intensive and usually not worth it compared to investing that time to get your app to run smoothly.

Yes but when someone else causes your downtime it’s fun to sit around and snipe at them for fun.

Re: AWS us-east-2 outage

#208

Post by rr888 »

poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

My take is that so many sites are broken, maybe I shouldn't care either. The extra complexity of dealing with high availability is something that probably isn't worth it for my project. Spend more time on features instead.

Re: AWS us-east-2 outage

#209

Post by throwbigdata »

jedberg wrote: post #162
Tehnix wrote: post #120

Earlier quoted context omitted.

> each AZ is considered a separate datacenter but in real life each AZ is multiple datacenters that are really close to each other For AWS specifically, I’m fairly certain they maintain a minimum distance and are much more strict on requirements to be on different grids etc than other Cloud providers. A few years ago they were calling out Azure and Google Cloud on exactly what you describe (having data centers essent…

A single AZ may have neighboring datacenters, but they are very strict on having datacenters for different AZs be at least 100km apart and on different flood plains and power grids.

100km? Oh really?

Re: AWS us-east-2 outage

#210

Post by throwbigdata »

water-your-self wrote: post #115
dangero wrote: post #101

Earlier quoted context omitted.

For most businesses a little down time here and there is a calculated risk versus more complex infrastructure. You can’t assume all the cloud architects are idiots — they have to report their task list and cost of infrastructure to someone who can give feedback on various options based on comparative resource requirements and risks. Zone downtime still falls under an AWS SLA so you know about how much downtime to acc…

It gives me a bad gut feeling when you imply that multiple instances of a service is more complex than a single instance which cannot be duplicated easily. I also disagree that it is inherently more costly to run a service in multiple locations.

I’m sorry about your feelings but you are wrong.

its more expensive to have more things and it’s more expensive to have more complicated things that are also complex. And things that can fall over are inherently more complicated.

Post reply on HN