Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS outage summary

aws.amazon.com

Post reply on HN
Print view View original

1–10 of 66 posts

Re: AWS outage summary

#2

Post by lukev »

I am always astonished by how many layers these bugs actually have. It's easy to start out blaming AWS, but if anyone can realistically say they could have anticipated this type of issue at a system level, they're deluding themselves.

Re: AWS outage summary

#4

Post by teraflop »

> Multi Availability Zone (Multi-AZ), where two database instances are synchronously operated in two different Availability Zones.

> The second group of Multi-AZ instances did not failover automatically because the master database instances were disconnected from their standby for a brief time interval immediately before these master database instances’ volumes became stuck. Normally these events are simultaneous. Between the period of time the masters were disconnected from their standbys and the point where volumes became stuck, the masters continued to process transactions without being able to replicate to their standbys.

Can someone explain this? I thought the entire point of synchronous replication was that the master doesn't acknowledge that a transaction is committed until the data reaches the slave. That's how it's described in the RDS FAQ: http://aws.amazon.com/rds/faqs/#36

Re: AWS outage summary

#5

Post by ndcrandall »

Everytime there is a service outage it makes me feel better about using them in the future. Every outage is actually making the project more reliable since some issuess will only manifest in production. I believe they have a great team that's very knowledgable.

Re: AWS outage summary

#6

Post by mcpherrinm »

teraflop wrote: post #4

> Multi Availability Zone (Multi-AZ), where two database instances are synchronously operated in two different Availability Zones. > The second group of Multi-AZ instances did not failover automatically because the master database instances were disconnected from their standby for a brief time interval immediately before these master database instances’ volumes became stuck. Normally these events are simultaneous. Be…

This just sounds like a race condition gone weird.

I assume from this description that the update protocol looks something like this:

   process-request:
     Update-local
     push to sync buddy
     wait for sync buddy success
     reply-with-status
     mark state as synced
If the function does the local update and then gets stuck in the state waiting for the buddy to reply, one could imagine the failover daemon not handling that case very well. So while the master might not have acknowledged the transaction, the pair might get jammed trying to complete it.

EBS is a particularily complicated piece of software, and RDS is another layer of complication built on top of that. Bugs clearly happen, and it's an unfortunate state of affairs.

Re: AWS outage summary

#7

Post by blaines »

ndcrandall wrote: post #5

Everytime there is a service outage it makes me feel better about using them in the future. Every outage is actually making the project more reliable since some issuess will only manifest in production. I believe they have a great team that's very knowledgable.

I don't necessarily agree with your first two sentences, but I definitely agree with the last one. I know they're smart.

I do have some concerns that they're having too much downtime. If there's one small flaw in the system it seems that the whole thing begins to fail.

If they fix the problem, and it impacts the larger system in some other unknown way, a different equally crippling issue could present itself in the future. I'd like to be sure they're putting a huge effort into making sure these problems don't happen, and I don't have those assurances at the moment.

First things first, a better status dashboard that actually reflects how issues impact customers is needed. I'd rather have everything working fine and the status be 'red' than have servers down, support tickets, calls, emails, etc and see a 'green' on the dashboard.

Re: AWS outage summary

#8

Post by papercruncher »

I know there are lots of smart people working there but just look at the sheer amount of AWS offerings. Amazon certainly gets credit for quickly putting out new features and services but it makes me wonder if their pace has resulted in way too many moving parts with an intractable number of dependencies.

Re: AWS outage summary

#9

Post by Trufa »

I really love when the companies take time to explain their customers what happened specially in such detail.

It's clearly a very complicated setting, and this type of posts make me trust them more, don't get me wrong, and outage is an outage, but knowing that they are in control and take time to explain shows respect and the correct attitude towards a mistake.

Good for them!

Post reply on HN