31–40 of 66 posts
AWS sure does put out amazing post mortems. If only they'd make their status page more useful ...
How is the status page not useful ? It's so simple to understand. Green if everything is fine. Green if there are intermittent problems. Green if nothing works.
> We are already in the process of making a few changes to reduce the interdependency between ELB and EBS to avoid correlated failure in future events and allow ELB recovery even when there are EBS issues within an Availability Zone. This is music to my ears. We switched away from ELBs because of this dependency. Hopefully this statement means Amazon is working on completely removing any use of EBS from ELBs. We came…
I am always astonished by how many layers these bugs actually have. It's easy to start out blaming AWS, but if anyone can realistically say they could have anticipated this type of issue at a system level, they're deluding themselves.
Full disclosure: I work for an AWS competitor. While none of the specific AWS systemic failures may themselves be foreseeable, it is not true that issues of this nature cannot be anticipated: the architecture of their system (and in particular, their insistence on network storage for local data) allows for cascading failure modes in which single failures blossom to systemic ones. AWS is not the only entity to have ma…
Earlier quoted context omitted.
Full disclosure: I work for an AWS competitor. While none of the specific AWS systemic failures may themselves be foreseeable, it is not true that issues of this nature cannot be anticipated: the architecture of their system (and in particular, their insistence on network storage for local data) allows for cascading failure modes in which single failures blossom to systemic ones. AWS is not the only entity to have ma…
Very wise comment IMHO. More complex = more complex failure modes. The only way to improve is Less complex. Well also probably local storage has a number of other advantages from the point of view of performance.
What feels wrong here, is that Amazon systems are too trusting to other Amazon internal systems.
I dunno about you, but I could use a TL;DR for this: 1. They fucked up an internal DNS change and didn't notice 2. Internal systems on EBS hosts piled up with messages trying to get to the non-existent domain 3. Eventually the messages used up all the memory on the EBS hosts, and thousands of EBS hosts began to die simultaneously 4. Meanwhile, panicked operators trying to slow down this tidal wave hit the Throttle Ev…
I dunno about you, but I could use a TL;DR for this: 1. They fucked up an internal DNS change and didn't notice 2. Internal systems on EBS hosts piled up with messages trying to get to the non-existent domain 3. Eventually the messages used up all the memory on the EBS hosts, and thousands of EBS hosts began to die simultaneously 4. Meanwhile, panicked operators trying to slow down this tidal wave hit the Throttle Ev…
I dunno about you, but I could use a TL;DR for this: 1. They fucked up an internal DNS change and didn't notice 2. Internal systems on EBS hosts piled up with messages trying to get to the non-existent domain 3. Eventually the messages used up all the memory on the EBS hosts, and thousands of EBS hosts began to die simultaneously 4. Meanwhile, panicked operators trying to slow down this tidal wave hit the Throttle Ev…
Earlier quoted context omitted.
Full disclosure: I work for an AWS competitor. While none of the specific AWS systemic failures may themselves be foreseeable, it is not true that issues of this nature cannot be anticipated: the architecture of their system (and in particular, their insistence on network storage for local data) allows for cascading failure modes in which single failures blossom to systemic ones. AWS is not the only entity to have ma…
I believe the damage of cascading failures was also masterfully exposed during your talk on QCon Last year: http://www.infoq.com/presentations/Debugging-Production-Syst...
Very entertaining IMO. But I would suggest the speaker cut the off-hand ostracism of large groups of people.
I dunno about you, but I could use a TL;DR for this: 1. They fucked up an internal DNS change and didn't notice 2. Internal systems on EBS hosts piled up with messages trying to get to the non-existent domain 3. Eventually the messages used up all the memory on the EBS hosts, and thousands of EBS hosts began to die simultaneously 4. Meanwhile, panicked operators trying to slow down this tidal wave hit the Throttle Ev…
My general take on their writeup was that they have too many services that depend on EBS working. They should try to find ways to decouple critical services like loadbalancers and Amazon RDS from EBS.
To your point, w/o knowing architecture, seems like ELB run state could likely be on ephemeral storage (if ELBs are EC2 instances) backed by configs on S3 unless run state is crucial across resets. If not instances, maybe use S3 directly, or ElastiCache.