Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

711–720 of 1001 posts

Re: AWS us-east-1 outage

#711

Post by lordnacho »

dylan604 wrote: post #657

The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…

If you did that some clever person would set up their PiHole so that their device just always worked, and then you couldn't send them ads and surveil them. They'd tell their friends and then everyone would just use their local devices locally. Totally irresponsible what you're suggesting.

Re: AWS us-east-1 outage

#712

Post by Rapzid »

joshocar wrote: post #513
amir734jj wrote: post #498

My job (although 50% of time) at Azure is unit testing/monitoring services under different scenarios and flows to detect small failures that will be overlooked in public status page. Our tests run multiple times daily and we have people constantly monitoring logs. It concerns me when I see all AWS services are 100% green when I know there is an outage.

I don't know how accurate this information is, but I'm hearing that the monitor can't be updated because the service is in the region that is down.

Kinda hard to believe after they were blasted for that very situation during/after the S3 outage way back.

If that's the case, it's 100% a feature. They want as little public proof of an outage after it's over and to put the burden on customers completely to prove they violated SLAs.

Re: AWS us-east-1 outage

#713

Post by ithkuil »

jesboat wrote: post #484
ipmb wrote: post #245

Looks like they've acknowledged it on the status page now. https://status.aws.amazon.com/ > 8:22 AM PST We are investigating increased error rates for the AWS Management Console. > 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US…

> This issue is affecting the global console landing page, which is also hosted in US-EAST-1 Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region? At a different (unnamed) FAANG, we considered it unacceptable to have anything depend on a single region. Even the dinky little volunteer-run thing which ran https://internal.site.example/~someEngineer was…

One region? I forgot how to count that low

Re: AWS us-east-1 outage

#714

Post by lesam »

jaywalk wrote: post #499
rossdavidh wrote: post #395

Earlier quoted context omitted.

If we have a total global outage, Stack Overflow will be unavailable, and the internet will never be fixed. :) Mostly joking, I hope...

Some brave soul at Stack Overflow will have to physically go into the datacenter, roll up a cart with a keyboard, monitor and printer and start printing off a bunch of Networking answers.

The StackOverflow datacenter is famously tiny - like, 10 Windows servers. So even if the rest of the internet goes down hopefully they stay up. They might have to rebuild the internet out from their server room though.

Re: AWS us-east-1 outage

#715

Post by simmanian »

rehevkor5 wrote: post #705
s_dev wrote: post #666

Earlier quoted context omitted.

>The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. Whats a graceful fallback? Switching to another hosting service when AWS goes down? Wouldn't that present another set of complications for a very small edge case at huge cost?

Usually this refers to falling back to a different region in AWS. It's typical for systems to be deployed in multiple regions due to latency concerns, but it's also important for resiliency. What you call "a very small edge case" is occurring as we speak, and if you're vulnerable to it you could be losing millions of dollars.

AWS itself has a huge single point of failure on us-east-1 region. Usually, if us-east-1 goes down, others soon follow. At that point, it doesn't matter how many regions you're deploying to.

Re: AWS us-east-1 outage

#716

Post by apexalpha »

lordnacho wrote: post #711
dylan604 wrote: post #657

The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…

If you did that some clever person would set up their PiHole so that their device just always worked, and then you couldn't send them ads and surveil them. They'd tell their friends and then everyone would just use their local devices locally. Totally irresponsible what you're suggesting.

A little off-topic, but there are people working on it: https://valetudo.cloud/

It's a little harder than blocking the DNS unfortunately. But nonetheless it always brings a smile to my face to see that there's a FOSS frontier for everything.

Re: AWS us-east-1 outage

#717

Post by beamatronic »

lordnacho wrote: post #711
dylan604 wrote: post #657

The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…

If you did that some clever person would set up their PiHole so that their device just always worked, and then you couldn't send them ads and surveil them. They'd tell their friends and then everyone would just use their local devices locally. Totally irresponsible what you're suggesting.

An even more clever person would package up this box, and sell it, along with a companion subscription service, to help busy folks like myself.

Re: AWS us-east-1 outage

#718

Post by baoyu »

tw04 wrote: post #694
boldman wrote: post #691

I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…

>It seems bad now but I wonder how much worse it might be when no one actually has access to money because all financial traffic is going through AWS and it goes down. Most financial institutions are implementing their own clouds, I can't think of any major one that is reliant on public cloud to the extent transactions would stop. >Why hasn't the industry come up with an alternative? You mean like building datacenter…

> Most financial institutions are implementing their own clouds

https://www.nasdaq.com/Nasdaq-AWS-cloud-announcement

Re: AWS us-east-1 outage

#719

Post by mrep »

tekromancr wrote: post #588
jesboat wrote: post #484

Earlier quoted context omitted.

> This issue is affecting the global console landing page, which is also hosted in US-EAST-1 Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region? At a different (unnamed) FAANG, we considered it unacceptable to have anything depend on a single region. Even the dinky little volunteer-run thing which ran https://internal.site.example/~someEngineer was…

I just want to serve 5 terabytes of data

Reference for those out of the loop: https://news.ycombinator.com/item?id=29082014

Re: AWS us-east-1 outage

#720

Post by sheenobu »

jesboat wrote: post #484
ipmb wrote: post #245

Looks like they've acknowledged it on the status page now. https://status.aws.amazon.com/ > 8:22 AM PST We are investigating increased error rates for the AWS Management Console. > 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US…

> This issue is affecting the global console landing page, which is also hosted in US-EAST-1 Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region? At a different (unnamed) FAANG, we considered it unacceptable to have anything depend on a single region. Even the dinky little volunteer-run thing which ran https://internal.site.example/~someEngineer was…

I think I know specifically what you are talking about. The actual files an engineer could upload to populate their folder was not multi-region for a long time. The servers were, because they were stateless and that was easy to multi-region, but the actual data wasn't until we replaced the storage service.
Post reply on HN