Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Service Interuptions

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 56 posts

Re: AWS Service Interuptions

#11

Post by sidcool »

From AWS status page for Asia Pacific:

10:47 PM PDT We are investigating increased connectivity issues for EC2 instances in the AP-SOUTHEAST-2 Region.

11:08 PM PDT We continue to investigate connectivity issues for some instances in a single Availability Zone and increased API error rates for the EC2 APIs in the AP-SOUTHEAST-2 Region.

11:49 PM PDT We can confirm that instances have experienced a power event within a single Availability Zone in the AP-SOUTHEAST-2 Region. Error rates for the EC2 APIs have improved and launches of new EC2 instances are succeeding within the other Availability Zones in the Region.

Jun 5, 12:31 AM PDT We have restored power to the affected Availability Zone and are working to restore connectivity to the affected instances.

Re: AWS Service Interuptions

#12

Post by madeofpalk »

sidcool wrote: post #11

From AWS status page for Asia Pacific: 10:47 PM PDT We are investigating increased connectivity issues for EC2 instances in the AP-SOUTHEAST-2 Region. 11:08 PM PDT We continue to investigate connectivity issues for some instances in a single Availability Zone and increased API error rates for the EC2 APIs in the AP-SOUTHEAST-2 Region. 11:49 PM PDT We can confirm that instances have experienced a power event within a…

Been having issues from Sydney AP-Southeast-2, probs from bigger-than-usual storm that's been going on here for the past few days.

Re: AWS Service Interuptions

#13

Post by lobe »

Not sure if relevant to this issue, but Sydney is currently being hit with one of the biggest storms I can remember in the past few years. Probably not crazy enough to take down a DC, but might be a contributing factor in this outage.

Re: AWS Service Interuptions

#15

Post by tvmalsv »

sidcool wrote: post #11

From AWS status page for Asia Pacific: 10:47 PM PDT We are investigating increased connectivity issues for EC2 instances in the AP-SOUTHEAST-2 Region. 11:08 PM PDT We continue to investigate connectivity issues for some instances in a single Availability Zone and increased API error rates for the EC2 APIs in the AP-SOUTHEAST-2 Region. 11:49 PM PDT We can confirm that instances have experienced a power event within a…

It took them an hour to figure out that their connectivity issues were caused by losing power to an entire Availability Zone? Maybe they should add an alert for "AZ has no power" or put it on a dashboard...

I'm joking of course, but that's what ran through my mind while reading that timeline.

Re: AWS Service Interuptions

#17

Post by AdamJacobMuller »

tvmalsv wrote: post #15
sidcool wrote: post #11

From AWS status page for Asia Pacific: 10:47 PM PDT We are investigating increased connectivity issues for EC2 instances in the AP-SOUTHEAST-2 Region. 11:08 PM PDT We continue to investigate connectivity issues for some instances in a single Availability Zone and increased API error rates for the EC2 APIs in the AP-SOUTHEAST-2 Region. 11:49 PM PDT We can confirm that instances have experienced a power event within a…

It took them an hour to figure out that their connectivity issues were caused by losing power to an entire Availability Zone? Maybe they should add an alert for "AZ has no power" or put it on a dashboard... I'm joking of course, but that's what ran through my mind while reading that timeline.

Wasn't quite that simple. I lost connectivity to instances that did not reboot so I'm guessing it took out some network elements.

Re: AWS Service Interuptions

#18

Post by origami777 »

I realize that some systems may need to have all of their servers located close together in a single AZ. But barring that, if this took you offline, you should really consider spreading your instances across AZs. It's so easy there's no excuse not to do it.

Another thing to look into is EC2 Auto Recovery [1]. I don't know if this would've kicked in with today's event, but it's worth setting up as an extra safety net.

[1] https://aws.amazon.com/blogs/aws/new-auto-recovery-for-amazo...

edit: I'm basing this off the status page which indicated that only one AZ was impacted.

Re: AWS Service Interuptions

#19

Post by PebblesHD »

origami777 wrote: post #18

I realize that some systems may need to have all of their servers located close together in a single AZ. But barring that, if this took you offline, you should really consider spreading your instances across AZs. It's so easy there's no excuse not to do it. Another thing to look into is EC2 Auto Recovery [1]. I don't know if this would've kicked in with today's event, but it's worth setting up as an extra safety net.…

Sadly our use case (private data etc.) prevents us from leaving the local availability zone, meaning when it went down today we were left totally unavailable. The recovery itself is ongoing but our applications are resilient enough to detect the restored connections and automatically add themselves back into the cluster.

Re: AWS Service Interuptions

#20

Post by origami777 »

PebblesHD wrote: post #19
origami777 wrote: post #18

I realize that some systems may need to have all of their servers located close together in a single AZ. But barring that, if this took you offline, you should really consider spreading your instances across AZs. It's so easy there's no excuse not to do it. Another thing to look into is EC2 Auto Recovery [1]. I don't know if this would've kicked in with today's event, but it's worth setting up as an extra safety net.…

Sadly our use case (private data etc.) prevents us from leaving the local availability zone, meaning when it went down today we were left totally unavailable. The recovery itself is ongoing but our applications are resilient enough to detect the restored connections and automatically add themselves back into the cluster.

That's interesting. Is it an Australian regulation? Curious that they'd make it in such a way that the data had to reside in the same building/zone.
Post reply on HN