Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS multiple services outage in us-east-1

health.aws.amazon.com

Post reply on HN
Print view View original

661–670 of 1001 posts

Re: AWS multiple services outage in us-east-1

#661

Post by ibejoeb »

manishsharan wrote: post #606
ibejoeb wrote: post #577

This is just a silly anecdote, but every time a cloud provider blips, I'm reminded. The worst architecture I've ever encountered was a system that was distributed across AWS, Azure, and GCP. Whenever any one of them had a problem, the system went down. It also cost 3x more than it should.

You mean multi-cloud strategy ! You wanna know how you got here ? See the sales team from Google flew out an executive to NBA Finals, Azure Sales team flew out another executive to NFL superBowl and the AWS team flew out yet another executive to Wimbledon finals. And thats how you end up with multi-cloud strategy.

In this particular case, it was resume-oriented architecture (ROAr!) The original team really wanted to use all the hottest new tech. The management was actually rather unhappy, so the job was to pare that down to something more reliable.

Re: AWS multiple services outage in us-east-1

#663

Post by ferguess_k »

AnimeLife wrote: post #611
ibejoeb wrote: post #577

This is just a silly anecdote, but every time a cloud provider blips, I'm reminded. The worst architecture I've ever encountered was a system that was distributed across AWS, Azure, and GCP. Whenever any one of them had a problem, the system went down. It also cost 3x more than it should.

looks like very few get it right. A good system would have few minutes of blip when one cloud provider went down, which is a massive win compared to outages like this.

I think the problems are:

1) If you try to optimize in the beginning, you tend to fall into the over-optimization/engineering camp;

2) If you just let things go organically, you tend to fall into the big messy camp;

So the ideal way is to examine from time and time and re-architecture once the need arises. But few companies can afford that, unfortunately.

Re: AWS multiple services outage in us-east-1

#664

Post by seany »

masfuerte wrote: post #592
JCM9 wrote: post #487

US-East-1 is more than just a normal region. It also provides the backbone for other services, including those in other regions. Thus simply being in another region doesn’t protect you from the consistent us-east-1 shenanigans. AWS doesn’t talk about that much publicly, but if you press them they will admit in private that there are some pretty nasty single points of failure in the design of AWS that can materialize…

Amazon are planning to launch the EU Sovereign Cloud by the end of the year. They claim it will be completely independent. It may be possible then to have genuine resiliency on AWS. We'll see.

gov, iso*, cn are also already separate (unless you need to mess with your bill, or certain kinds of support tickets)

Re: AWS multiple services outage in us-east-1

#665

Post by cyberax »

runako wrote: post #494

Even though us-east-1 is the region geographically closest to me, I always choose another region as default due to us-east-1 (seemingly) being more prone to these outages. Obviously, some services are only available in us-east-1, but many applications can gain some resiliency just by making a primary home in any other region.

This is the right move. 10 years ago, us-east-1 was on the order of 10x bigger than the next largest region. This got a little better now, but any scaling issues still tend to happen in us-east-1.

The AWS has been steering people to us-east-2 for a while. For example, traffic between us-east-1 and us-east-2 has the same cost as inter-AZ traffic within the us-east-1.

Re: AWS multiple services outage in us-east-1

#668

Post by lapetitejort »

1970-01-01 wrote: post #614

Someone, somewhere, had to report that doorbells went down because the very big cloud did not stay up. I think we're doing the 21st century wrong.

Someone, somewhere, had to report that rock throwers went down because the very big cloud did not stay up.

I think we're doing the 16th century wrong.

Re: AWS multiple services outage in us-east-1

#669

Post by freedomben »

mentalgear wrote: post #576

> Amazon Alexa: routines like pre-set alarms were not functioning. It's ridiculous how everything is being stored in the cloud, even simple timers. It's past high time to move functionality back on-device, which would come with the advantage of making it easier to de-connect from big tech's capitalist surveillance state as well.

Exactly.

I half-seriously like to say things like, "I'm excited for a time when we have powerful enough computers to actually run applications on them instead of being limited to only thin clients." Only problem is most of the younger people don't get the reference anymore, so it's mainly the olds that get it

Post reply on HN