AWS EC2/RDS Outage in us-east-1 • Hacker News Board

Re: AWS EC2/RDS Outage in us-east-1

#61

Post by matt2000 » Sat, Aug 31, 2019, 4:11 PM UTC

Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run.

I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ with a straightforward failover option is usually the right tradeoff.

Re: AWS EC2/RDS Outage in us-east-1

#62

Post by sjwright » Sat, Aug 31, 2019, 4:12 PM UTC

neuromantik8086 wrote: post #56

somehowadev wrote: post #33

I’m surprised by how much of the “internet” seem to be affected by a single AZ going down.

We wouldn't have this problem if people just used application-layer protocols and federated services like the early internet.

Wait, why wouldn’t we have these problems? Back in the 1980s, if a university campus connection goes down, you can’t telnet in or read your university POP2 email remotely. It’s down.

The only difference between then and now is that we’re online (seemingly) at every waking minute expecting a hundred different services to be functional at any given moment.

Re: AWS EC2/RDS Outage in us-east-1

#63

Post by jabart » Sat, Aug 31, 2019, 4:14 PM UTC

nrxr wrote: post #27

Has anyone else noticed that there seems to never be outages in us-east-2 and somehow everyone keeps putting instances in -1? Why?

us-east-2 has had a 2-3 hour issue with their internet routes which affected all of us-east-2. 2019 so far us-east-2 has been fine for me. My colo has had only 3 5minute outages this year. Keeping the cloud up is tough if even AWS can have large outages still.

Re: AWS EC2/RDS Outage in us-east-1

#64

Post by scott113341 » Sat, Aug 31, 2019, 4:15 PM UTC

I got paged 50 minutes before AWS updated their status page. We are running on AWS's managed Kubernetes offering (EKS), and about one third of our nodes were running in the affected availability zone. We were then able to move all of or traffic out of that AZ, which solved our issues. The main symptom was HTTP requests made by our backend to 3rd party APIs failing, but only on requests originating from that AZ.

Re: AWS EC2/RDS Outage in us-east-1

#65

Post by yjftsjthsd-h » Sat, Aug 31, 2019, 4:15 PM UTC

juliusmusseau wrote: post #60

dekhn wrote: post #55

Earlier quoted context omitted.

usually, DB servers will live in a small number of locations with good connectivity between clusters and the frontends (which terminate the user's TCP connection) live much closer (likely Sydney). Good design means that there are few roundtrips between the FEs and the backend but they are not unavoidable. Designing truly resilient and available applications with DB servers that replicate across continents is hard.

Is true master-master replication across continents even possible? I guess partitioning can help, but then isn't it just turning the DB servers into pizzas of master-slave where the Hawaiian slice is master only in Hawaii, and slave everywhere else?

Yeah, you're gonna hit CAP hard at that distance.

Re: AWS EC2/RDS Outage in us-east-1

#66

Post by bob1029 » Sat, Aug 31, 2019, 4:16 PM UTC

nrxr wrote: post #27

Has anyone else noticed that there seems to never be outages in us-east-2 and somehow everyone keeps putting instances in -1? Why?

For us, it's mostly a matter of historical convention. Our entire stack currently lives in -1 (we've had instances there for ~5 years now), and to move to a different region under these pretenses is a bit of a pain in the ass for us considering how transient the impact of these things has typically been to our business.

If we move anywhere, its going to be completely out of AWS and into on-prem or some bare metal provider. Hopping regions hoping to win at some reliability metric game is not a good way to run a business IMO.

Re: AWS EC2/RDS Outage in us-east-1

#67

Post by eternalny1 » Sat, Aug 31, 2019, 4:17 PM UTC

groundlogic wrote: post #2

Reddit has been quite dysfunctional for me the past hour or so.

Yes unable to login, noticed it right aware as the bright white glare blasted through what would otherwise be night mode.

Re: AWS EC2/RDS Outage in us-east-1

#68

Post by bdcravens » Sat, Aug 31, 2019, 4:20 PM UTC

Matthias247 wrote: post #51

bdcravens wrote: post #18

Earlier quoted context omitted.

Also, it is one of the regions that gets new features first, which makes me wonder if it contributes to lower stability.

This is not true. The region where new software is deployed first is different team by team (or service by service).

Perhaps, but I can't recall a product launch that wasn't available in us-east-1 from day 1.

Re: AWS EC2/RDS Outage in us-east-1

#69

Post by DVassallo » Sat, Aug 31, 2019, 4:21 PM UTC

nrxr wrote: post #27

Has anyone else noticed that there seems to never be outages in us-east-2 and somehow everyone keeps putting instances in -1? Why?

Because the outages there don’t make the HN frontpage.

Re: AWS EC2/RDS Outage in us-east-1

#70

Post by jen_h » Sat, Aug 31, 2019, 4:30 PM UTC

matt2000 wrote: post #61

Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…

An occasional outage is sometimes good for an app (depending, of course, on how mission-critical it is):

1. People don't realize how much they love and depend on you until you're gone.

2. Keeps you on your toes, it's easy to get complacent when everything just runs along happily for months and years on end.

I do wish there was a way to train users that millions of them reloading constantly as service ramps back up doesn't accelerate the ramp-up time, though. ;)