Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS outage summary

aws.amazon.com

Post reply on HN
Print view View original

21–30 of 66 posts

Re: AWS outage summary

#21

Post by kordless »

signifiers wrote: post #19
taligent wrote: post #13

Earlier quoted context omitted.

How is the status page not useful ? It's so simple to understand. Green if everything is fine. Green if there are intermittent problems. Green if nothing works.

Too true. Today when Google's App Engine failed hard, they flat out said: "App Engine is currently experiencing serving issues – Python, Java, Go … Oct 26 2012, 07:30 AM-11:59 PM … Current Availability: 55.90%" and showed read/write I/O metrics were spiking through the roof. Their dashboard clearly labeled the relevant icon as the most severe "Service Interruption". I admire that level of transparency.

Yeah, was thinking the same thing while looking at all the cool graphs.

Re: AWS outage summary

#23

Post by bigiain »

bcantrill wrote: post #15
lukev wrote: post #2

I am always astonished by how many layers these bugs actually have. It's easy to start out blaming AWS, but if anyone can realistically say they could have anticipated this type of issue at a system level, they're deluding themselves.

Full disclosure: I work for an AWS competitor. While none of the specific AWS systemic failures may themselves be foreseeable, it is not true that issues of this nature cannot be anticipated: the architecture of their system (and in particular, their insistence on network storage for local data) allows for cascading failure modes in which single failures blossom to systemic ones. AWS is not the only entity to have ma…

FWIW, once Amazon decided that an availability zone is an unreliable unit (and they tell you this up front, and strongly suggest running multiple AZ architectures for anything where you require reliability), then any cascading failure mode in a single AZ is not something you'd expect them to spend too much time protecting against. Sure, the cascade from EBS faults to RDS and ELB meant this affected more of their single AZ customers than it would have otherwise, but anyone using a singe AZ knew upfront that Amazon advised against that, and never claimed they intended to provide high-availability in single AZs.

So yeah, you're right - these systemic failures can be anticipated, and Amazon's advice (to spread your important infrastructure across multiple AZs) would have protected users from most of this. (I feel quite a lot of sympathy for the engineers involved in the cross-AZ failures this incident revealed - the multi-AZ RDS and ELB failures are things customers doing everything they were told got bitten by anyway, and are probably rightly annoyed...)

Re: AWS outage summary

#24

Post by bigiain »

taligent wrote: post #13
mrkurt wrote: post #10

AWS sure does put out amazing post mortems. If only they'd make their status page more useful ...

How is the status page not useful ? It's so simple to understand. Green if everything is fine. Green if there are intermittent problems. Green if nothing works.

You left out "Green if everything is working - except the Chaos Monkey. All systems are working properly, but none of them failing randomly 'for your convenience'"

Re: AWS outage summary

#25

Post by antirez »

bcantrill wrote: post #15
lukev wrote: post #2

I am always astonished by how many layers these bugs actually have. It's easy to start out blaming AWS, but if anyone can realistically say they could have anticipated this type of issue at a system level, they're deluding themselves.

Full disclosure: I work for an AWS competitor. While none of the specific AWS systemic failures may themselves be foreseeable, it is not true that issues of this nature cannot be anticipated: the architecture of their system (and in particular, their insistence on network storage for local data) allows for cascading failure modes in which single failures blossom to systemic ones. AWS is not the only entity to have ma…

Very wise comment IMHO. More complex = more complex failure modes. The only way to improve is Less complex. Well also probably local storage has a number of other advantages from the point of view of performance.

Re: AWS outage summary

#26

Post by redditmigrant »

teraflop wrote: post #4

> Multi Availability Zone (Multi-AZ), where two database instances are synchronously operated in two different Availability Zones. > The second group of Multi-AZ instances did not failover automatically because the master database instances were disconnected from their standby for a brief time interval immediately before these master database instances’ volumes became stuck. Normally these events are simultaneous. Be…

I think aws uses drbd (http://www.drbd.org/) for replicating the writes to the standby. Among the replication modes of drbd suggested here: http://www.drbd.org/users-guide-emb/s-replication-protocols.... I suspect they probably use B and landed in a state where packets got delivered but werent written on the standby.

Re: AWS outage summary

#27

Post by mbesto »

blaines wrote: post #7
ndcrandall wrote: post #5

Everytime there is a service outage it makes me feel better about using them in the future. Every outage is actually making the project more reliable since some issuess will only manifest in production. I believe they have a great team that's very knowledgable.

I don't necessarily agree with your first two sentences, but I definitely agree with the last one. I know they're smart. I do have some concerns that they're having too much downtime. If there's one small flaw in the system it seems that the whole thing begins to fail. If they fix the problem, and it impacts the larger system in some other unknown way, a different equally crippling issue could present itself in the f…

Do you think maybe this is just a focusing illusion?[1] And therefore the utility you associate with the service is not correctly attributed.

[1] http://en.wikipedia.org/wiki/Anchoring

Re: AWS outage summary

#28

Post by seldo »

I dunno about you, but I could use a TL;DR for this:

1. They fucked up an internal DNS change and didn't notice

2. Internal systems on EBS hosts piled up with messages trying to get to the non-existent domain

3. Eventually the messages used up all the memory on the EBS hosts, and thousands of EBS hosts began to die simultaneously

4. Meanwhile, panicked operators trying to slow down this tidal wave hit the Throttle Everything button

5. The throttling was so aggressive the even normal levels of operation became impossible

6. The incident was a single AZ, but the throttling was across the whole region, which spread the pain further

[Everybody who got throttled gets a 3-hour refund]

7. Any single-AZ RDS instance on a dead EBS host was fucked

8. Multi-AZ RDS instances ran into two separate bugs, and either became stuck or hit a replication race condition and shut down

[Everybody whose multi-AZ RDS didn't fail over gets 10 days free credit]

9. Single-AZ ELB instances in the broken AZ failed because they use EBS too

10. Because everybody was freaking out and trying to fix their ELBs, the ELB service ran out of IP addresses and locked up

11. Multi-AZ ELB instances took too long to notice EBS was broken and then hit a bug and didn't fail over properly anyway

[ELB users get no refund, which seems harsh]

For those keeping score, that's 1 human error, 2 dependency chains, 3 design flaws, 3 instances of inadequate monitoring, and 5 brand-new internal bugs. From the length and groveling tone of the report, I can only assume that a big chunk of customers are very, VERY angry at them.

Re: AWS outage summary

#29

Post by krosaen »

Bugs happen, and the effects of cascading failures are very hard to anticipate. But it seems like the aws team hadn't fully tested the effects of an EBS outage, which seems like it could have uncovered the rdms multi availability zone failover bug and perhaps the elbs failover bug ahead of time.

Re: AWS outage summary

#30

Post by seldo »

bigiain wrote: post #23
bcantrill wrote: post #15

Earlier quoted context omitted.

Full disclosure: I work for an AWS competitor. While none of the specific AWS systemic failures may themselves be foreseeable, it is not true that issues of this nature cannot be anticipated: the architecture of their system (and in particular, their insistence on network storage for local data) allows for cascading failure modes in which single failures blossom to systemic ones. AWS is not the only entity to have ma…

FWIW, once Amazon decided that an availability zone is an unreliable unit (and they tell you this up front, and strongly suggest running multiple AZ architectures for anything where you require reliability), then any cascading failure mode in a single AZ is not something you'd expect them to spend too much time protecting against. Sure, the cascade from EBS faults to RDS and ELB meant this affected more of their sing…

While Amazon do indeed say that multi-AZ is the way to go, their last 3 major incidents (including last year's cloudpocalypse) have all been full-region incidents.

IMHO, their biggest design problem is that they build their systems on top of each other (e.g. ELB is built on EBS and EIP). So when one system goes down, it takes down half a dozen others -- this is especially true of EBS, and especially dangerous because the services it takes down, like ELB, are the services people are supposed to be using to route around EBS failures.

Post reply on HN