Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS multiple services outage in us-east-1

health.aws.amazon.com

Post reply on HN
Print view View original

561–570 of 1001 posts

Re: AWS multiple services outage in us-east-1

#561

Post by avi_vallarapu »

This is the reason why it is important to plan Disaster recovery and also plan Multi-Cloud architectures.

Our applications and databases must have ultra high availability. It can be achieved with applications and data platforms hosted on different regions for failover.

Critical businesses should also plan for replication across multiple cloud platforms. You may use some of the existing solutions out there that can help with such implementations for data platforms.

- Qlik replicate - HexaRocket

and some more.

Or rather implement native replication solutions available with data platforms.

Re: AWS multiple services outage in us-east-1

#562

Post by Thaxll »

pjmlp wrote: post #485

It just goes to show the difference between best practices in cloud computing, and what everyone ends up doing in reality, including well known industry names.

Best practice does not include plan for when AWS going down. Netflix does not plan for it and they have a very strong eng org.

Re: AWS multiple services outage in us-east-1

#563

Post by runako »

joncrane wrote: post #511
runako wrote: post #494

Even though us-east-1 is the region geographically closest to me, I always choose another region as default due to us-east-1 (seemingly) being more prone to these outages. Obviously, some services are only available in us-east-1, but many applications can gain some resiliency just by making a primary home in any other region.

What services are only available in us-east-1?

In addition to those listed in sibling comments, new services often roll out in us-east-1 before being made available in other regions.

I recently ran into an issue where some Bedrock functionality was available in us-east-1 but not one of the other US regions.

Re: AWS multiple services outage in us-east-1

#565

Post by maerF0x0 »

padjo wrote: post #369
jacquesm wrote: post #353

Every week or so we interview a company and ask them if they have a fall-back plan in case AWS goes down or their cloud account disappears. They always have this deer-in-the-headlights look. 'That can't happen, right?' Now imagine for a bit that it will never come back up. See where that leads you. The internet got its main strengths from the fact that it was completely decentralized. We've been systematically erodin…

Planning for an AWS outage is a complete waste of time and energy for most companies. Yes it does happen but very rarely to the tune of a few hours every 5-10 years. I can almost guarantee that whatever plans you have won’t get you fully operational faster than just waiting for AWS to fix it.

Using AWS instead of a server in the closet is step 1.

Step 2 is multi-AZ

Step 3 is multi-region

Step 4 is multi-cloud.

Each company can work on it's next step, but most will not have positive EROI going from 2 to 3+

Re: AWS multiple services outage in us-east-1

#566

Post by notahacker »

comrade1234 wrote: post #313

I like that we can advertise to our customers that over the last X years we have better uptime than Amazon, google, etc.

Feel bad for the Amazon SDR randomly pitching me AWS services today. Although apparently our former head of marketing got that pitch from four different LinkedIn accounts. Maybe there's a cloud service to rein them in that broke ;)

Re: AWS multiple services outage in us-east-1

#567

Post by dangoldin »

padjo wrote: post #369
jacquesm wrote: post #353

Every week or so we interview a company and ask them if they have a fall-back plan in case AWS goes down or their cloud account disappears. They always have this deer-in-the-headlights look. 'That can't happen, right?' Now imagine for a bit that it will never come back up. See where that leads you. The internet got its main strengths from the fact that it was completely decentralized. We've been systematically erodin…

Planning for an AWS outage is a complete waste of time and energy for most companies. Yes it does happen but very rarely to the tune of a few hours every 5-10 years. I can almost guarantee that whatever plans you have won’t get you fully operational faster than just waiting for AWS to fix it.

I worked at an adtech company where we invested a bit in HA across AZ + regions. Lo and behold there was an AWS outage and we stayed up. Too bad our customers didn't and we still took the revenue hit.

Lesson here is that your approach will depend on your industry and peers. Every market will have their won philosophy and requirements here.

Re: AWS multiple services outage in us-east-1

#568

Post by qaq »

JCM9 wrote: post #487

US-East-1 is more than just a normal region. It also provides the backbone for other services, including those in other regions. Thus simply being in another region doesn’t protect you from the consistent us-east-1 shenanigans. AWS doesn’t talk about that much publicly, but if you press them they will admit in private that there are some pretty nasty single points of failure in the design of AWS that can materialize…

even if us-east-1 was a normal region there is not enough spare capacity to take up all the workloads from us-east-1 in other regions so t's a moot point

Re: AWS multiple services outage in us-east-1

#569

Post by aurareturn »

api wrote: post #551
JCM9 wrote: post #487

US-East-1 is more than just a normal region. It also provides the backbone for other services, including those in other regions. Thus simply being in another region doesn’t protect you from the consistent us-east-1 shenanigans. AWS doesn’t talk about that much publicly, but if you press them they will admit in private that there are some pretty nasty single points of failure in the design of AWS that can materialize…

My contention for a long time has been that cloud is full of single points of failure (and nightmarish security hazards) that are just hidden from the customer. "We can't run things on just a box! That's a single point of failure. We're moving to cloud!" The difference is that when the cloud goes down you can shift the blame to them, not you, and fixing it is their problem. The corporate world is full of stuff like t…

Or it is a matter of efficiency. If 1 million companies design and maintain their servers, there would be 1 million (or more) incidents like these. Same issues. Same fixes. Not so efficient.

Re: AWS multiple services outage in us-east-1

#570

Post by aurareturn »

rose-knuckle17 wrote: post #555

aws had an outage. Many companies were impacted. Headlines around the world blame AWS. the real news is how easy it is to identify companies that have put cost management ahead of service resiliency. Lots of orgs operating wholly in AWS and sometimes only within us-east-1 had no operational problems last night. Some that is design (not using the impacted services). Some of that is good resiliency in design. And some…

Redundancy is insanely expensive especially for SaaS companies where the biggest cost is cloud.

Are customers willing to pay companies for that redundancy? I think not. Once every few years outage for 3 hours is fine for non critical services.

Post reply on HN