Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Service Interuptions

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 56 posts

Re: AWS Service Interuptions

#21

Post by PebblesHD »

origami777 wrote: post #20
PebblesHD wrote: post #19

Earlier quoted context omitted.

Sadly our use case (private data etc.) prevents us from leaving the local availability zone, meaning when it went down today we were left totally unavailable. The recovery itself is ongoing but our applications are resilient enough to detect the restored connections and automatically add themselves back into the cluster.

That's interesting. Is it an Australian regulation? Curious that they'd make it in such a way that the data had to reside in the same building/zone.

Indeed it is, it was a massive struggle getting approval to move into a cloud service in the first place.

Re: AWS Service Interuptions

#22

Post by [deleted] »

origami777 wrote: post #18

I realize that some systems may need to have all of their servers located close together in a single AZ. But barring that, if this took you offline, you should really consider spreading your instances across AZs. It's so easy there's no excuse not to do it. Another thing to look into is EC2 Auto Recovery [1]. I don't know if this would've kicked in with today's event, but it's worth setting up as an extra safety net.…

[deleted]

Re: AWS Service Interuptions

#26

Post by rnhu »

origami777 wrote: post #18

I realize that some systems may need to have all of their servers located close together in a single AZ. But barring that, if this took you offline, you should really consider spreading your instances across AZs. It's so easy there's no excuse not to do it. Another thing to look into is EC2 Auto Recovery [1]. I don't know if this would've kicked in with today's event, but it's worth setting up as an extra safety net.…

The site I manage is load balanced across both AZs ap-southeast-2a and ap-southeast-2b which did not save it. At the moment ec2 statuses are not being updated which is preventing ELBs from registering instances as healthy.

Both AZs are directly under the deluge and I don't believe only one AZ is affected for a second.

The size of the storm can be seen here http://www.bom.gov.au/products/IDR713.loop.shtml#skip

Re: AWS Service Interuptions

#27

Post by NeutronBoy »

PebblesHD wrote: post #19
origami777 wrote: post #18

I realize that some systems may need to have all of their servers located close together in a single AZ. But barring that, if this took you offline, you should really consider spreading your instances across AZs. It's so easy there's no excuse not to do it. Another thing to look into is EC2 Auto Recovery [1]. I don't know if this would've kicked in with today's event, but it's worth setting up as an extra safety net.…

Sadly our use case (private data etc.) prevents us from leaving the local availability zone, meaning when it went down today we were left totally unavailable. The recovery itself is ongoing but our applications are resilient enough to detect the restored connections and automatically add themselves back into the cluster.

Availability zones are different from regions. You can be in multiple AZ's within the Sydney region still.

Re: AWS Service Interuptions

#28

Post by mrmondo »

romanr wrote: post #24

It will take a direct hit with nuclear weapon on the datacenter for Amazon to change icon to red on service status page.

Yeah we monitor lots of Amazon & Microsoft 'cloud' services, we observe much, much higher downtime / number of outages than they ever report in a order of 50 to 1 or more. What do you expect though, both companies are known for lying through teeth to convince the IT community (or more likely the IT managers) that their services are reliable for everyone and amazing uptime and that they're not only a good option but the only option.

Re: AWS Service Interuptions

#29

Post by [deleted] »

NeutronBoy wrote: post #27
PebblesHD wrote: post #19

Earlier quoted context omitted.

Sadly our use case (private data etc.) prevents us from leaving the local availability zone, meaning when it went down today we were left totally unavailable. The recovery itself is ongoing but our applications are resilient enough to detect the restored connections and automatically add themselves back into the cluster.

Availability zones are different from regions. You can be in multiple AZ's within the Sydney region still.

[deleted]

Re: AWS Service Interuptions

#30

Post by inopinatus »

PebblesHD wrote: post #21
origami777 wrote: post #20

Earlier quoted context omitted.

That's interesting. Is it an Australian regulation? Curious that they'd make it in such a way that the data had to reside in the same building/zone.

Indeed it is, it was a massive struggle getting approval to move into a cloud service in the first place.

I almost hate to point this out, then, but you did consider that there's no guarantee that an AZ is a single DC, right?
Post reply on HN