Summary of the AWS Service Event in the Sydney Region • Hacker News Board

Re: Summary of the AWS Service Event in the Sydney Region

#31

Post by jread » Thu, Jun 09, 2016, 5:56 PM UTC

brazzledazzle wrote: post #29

jread wrote: post #27

Earlier quoted context omitted.

I independently monitor availability of 150 public cloud services and only observed 1.73 hours downtime for this event. This is the first EC2 outage I've observed in any region for over 6 months. According to my stats, in 2015 EC2 was highly available with 6 of 9 regions (including every US and EU region) having no outages, and total average service availability of 99.998% (78% of downtime in sa-east-1). I haven't ob…

Most of the criticism seemed to be centered around communication/PR so statistics don't really address that. Definitely important when considering a provider though. While the numbers are nice, if some outages only impact a subset of customers and your monitoring accounts aren't one of them it's hard to determine how good your monitoring data really is. If he was impacted by a 12 hour outage and you only show ~2 hour…

2 hours aligns with the postmortem, 12 hours does not. The monitoring is based on a sampling of instances running in each service and region. Outages are verified from multiple network paths. While not comprehensive, over the years this has been generally accurate because most outage events have been network or power related, thus impacting most or all instances in the affected data center. There have been some isolated hardware failures, but they are rare.

Re: Summary of the AWS Service Event in the Sydney Region

#32

Post by atonse » Thu, Jun 09, 2016, 5:56 PM UTC

brazzledazzle wrote: post #30

beachstartup wrote: post #22

Earlier quoted context omitted.

people vote with their dollars and their votes are overwhelmingly telling amazon they're doing a great job.

For the record I agree with you but existing customers who are heavily invested in AWS would find it difficult to vote with their dollars.

Agreed. I have a couple of clients that have a pretty substantial AWS spend, but the cost of switching to Azure is too high compared to the difference in offerings. You don't want to spend tens of thousands of dollars of developer time and risk switching datacenters for a small improvement.

Re: Summary of the AWS Service Event in the Sydney Region

#33

Post by jdc0589 » Thu, Jun 09, 2016, 6:35 PM UTC

shermozle wrote: post #13

I'm a bit dubious about their "if you used multi-AZ you'll be fine" when I had multiple outages in a multi-AZ Elastic Beanstalk application of over an hour. Methinks the load balancers aren't as magical as they'd like to make out.

> Methinks the load balancers aren't as magical as they'd like to make out.

Agreed. I'm still put off by the fact that ELBs specifically can not handle a sudden spike in traffic orders of magnitude higher than the previous rate. They fall on their face, bad. If you expect a spike like that to happen, you literally have to submit a ticket and ask them to pre-warm your ELB...

Re: Summary of the AWS Service Event in the Sydney Region

#34

Post by beachstartup » Thu, Jun 09, 2016, 6:48 PM UTC

brazzledazzle wrote: post #30

beachstartup wrote: post #22

Earlier quoted context omitted.

people vote with their dollars and their votes are overwhelmingly telling amazon they're doing a great job.

For the record I agree with you but existing customers who are heavily invested in AWS would find it difficult to vote with their dollars.

yeah, it works out great for amazon. less work, more money, what's not to love.

Re: Summary of the AWS Service Event in the Sydney Region

#35

Post by plandis » Thu, Jun 09, 2016, 7:21 PM UTC

jaketay wrote: post #15

Our instances with ap-southeast-2 were out for around 12 hours. We used multiple availability zones and it didn't prevent downtime at all. It's very interesting the difference between AWS and Google outage responses. AWS is down for 12+ hours for some customers, force each customer to chase service level credits and sign off the postmortem with a nameless & faceless "-The AWS Team". Not one person at AWS was willing…

Do you have proof? Your claims directly contradict what was written in their post mortem both in terms of time and scope. AWS might be using nice wording but I can't see them lying that much about the scope of the incident.

Re: Summary of the AWS Service Event in the Sydney Region

#36

Post by jread » Thu, Jun 09, 2016, 7:33 PM UTC

brazzledazzle wrote: post #29

jread wrote: post #27

Earlier quoted context omitted.

I independently monitor availability of 150 public cloud services and only observed 1.73 hours downtime for this event. This is the first EC2 outage I've observed in any region for over 6 months. According to my stats, in 2015 EC2 was highly available with 6 of 9 regions (including every US and EU region) having no outages, and total average service availability of 99.998% (78% of downtime in sa-east-1). I haven't ob…

Most of the criticism seemed to be centered around communication/PR so statistics don't really address that. Definitely important when considering a provider though. While the numbers are nice, if some outages only impact a subset of customers and your monitoring accounts aren't one of them it's hard to determine how good your monitoring data really is. If he was impacted by a 12 hour outage and you only show ~2 hour…

The stats are simply to provide some context. I agree criticism is centered around communication/PR, but also seems to imply the outage was of the same magnitude or worse than GCE's global outage, which is inaccurate. Per HA best practices, multi-region load balancing or failover would have adverted downtime during this event, whereas it would not have during the global GCE outage. Had this been a global outage, I think the postmortem and customer outreach would have been much better.

Re: Summary of the AWS Service Event in the Sydney Region

#37

Post by brazzledazzle » Thu, Jun 09, 2016, 7:58 PM UTC

jread wrote: post #31

brazzledazzle wrote: post #29

Earlier quoted context omitted.

Most of the criticism seemed to be centered around communication/PR so statistics don't really address that. Definitely important when considering a provider though. While the numbers are nice, if some outages only impact a subset of customers and your monitoring accounts aren't one of them it's hard to determine how good your monitoring data really is. If he was impacted by a 12 hour outage and you only show ~2 hour…

2 hours aligns with the postmortem, 12 hours does not. The monitoring is based on a sampling of instances running in each service and region. Outages are verified from multiple network paths. While not comprehensive, over the years this has been generally accurate because most outage events have been network or power related, thus impacting most or all instances in the affected data center. There have been some isola…

I may be mistaken, but 2 hours only aligns with their post mortem for 80% of the impacted instances. By their own account there were instances impacted until 8AM and a small number after that even.

Your monitoring sounds really comprehensive. That's a very cool way to advertise the service it's built on. Do you monitor service providers for outages that reduce capacity or increase latencies but otherwise the service is "up"?

Re: Summary of the AWS Service Event in the Sydney Region

#38

Post by brazzledazzle » Thu, Jun 09, 2016, 8:02 PM UTC

jread wrote: post #36

brazzledazzle wrote: post #29

Earlier quoted context omitted.

Most of the criticism seemed to be centered around communication/PR so statistics don't really address that. Definitely important when considering a provider though. While the numbers are nice, if some outages only impact a subset of customers and your monitoring accounts aren't one of them it's hard to determine how good your monitoring data really is. If he was impacted by a 12 hour outage and you only show ~2 hour…

The stats are simply to provide some context. I agree criticism is centered around communication/PR, but also seems to imply the outage was of the same magnitude or worse than GCE's global outage, which is inaccurate. Per HA best practices, multi-region load balancing or failover would have adverted downtime during this event, whereas it would not have during the global GCE outage. Had this been a global outage, I th…

You're right. It's not quite as bad as apples to oranges but comparing them without contextualizing is hardly fair.

Re: Summary of the AWS Service Event in the Sydney Region

#39

Post by thesandlord » Thu, Jun 09, 2016, 11:14 PM UTC

jdc0589 wrote: post #33

shermozle wrote: post #13

I'm a bit dubious about their "if you used multi-AZ you'll be fine" when I had multiple outages in a multi-AZ Elastic Beanstalk application of over an hour. Methinks the load balancers aren't as magical as they'd like to make out.

> Methinks the load balancers aren't as magical as they'd like to make out. Agreed. I'm still put off by the fact that ELBs specifically can not handle a sudden spike in traffic orders of magnitude higher than the previous rate. They fall on their face, bad. If you expect a spike like that to happen, you literally have to submit a ticket and ask them to pre-warm your ELB...

Yeah this is one big advantage GCP has over AWS, no need to prewarm load balancers. Not sure how Azure works (I work for Google).

Re: Summary of the AWS Service Event in the Sydney Region

#40

Post by bigiain » Fri, Jun 10, 2016, 1:20 AM UTC

ejdyksen wrote: post #23

jaketay wrote: post #15

Our instances with ap-southeast-2 were out for around 12 hours. We used multiple availability zones and it didn't prevent downtime at all. It's very interesting the difference between AWS and Google outage responses. AWS is down for 12+ hours for some customers, force each customer to chase service level credits and sign off the postmortem with a nameless & faceless "-The AWS Team". Not one person at AWS was willing…

> We used multiple availability zones and it didn't prevent downtime at all. Can you explain this a little more? Amazon says this only affected one AZ, and they specifically note: For this event, customers that were running their applications across multiple Availability Zones in the Region were able to maintain availability throughout the event.

+1.

Apart from one internal project which mistakenly had all it's app server instances in -2b (ooops!) - all my production mobile app backends are spread across the 3 Sydney AZs. That's a few dozen EC2 app servers across about 15 projects.

My monitoring reported a worst case of 57 seconds of degraded connectivity - which was an instance in -2b going offline and the ELB not taking it out of the rotation very quickly, the app running on that had interruption, but only while waiting for the timeouts. Crashlytics and GA crash reporting didn't bat an eyelid... I had under 70 users active at the time, 1/3rd of them may have seen a minute or less of loading spinner if they'd fired of a UI blocking api call during those 57 seconds. I'm not looking _super_ closely, but nothing I'm monitoring apart from EC2 - like RDS, S3, ELB, SNS - showed _any_ glitches (I'd _probably_ have caught even single digit second problems for _some_ of that...)

I'm actually quite happy with how everything went - we don't go to any particular heroic lengths to ensure HA or uptime, we just follow recommended best practice, and at least in this outage, that worked out fine for us (except for that project where all the app servers were in -2b, and I'm happy to wear that as our fuckup)