Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon DynamoDB Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

21–30 of 89 posts

Re: Summary of the Amazon DynamoDB Service Disruption

#21

Post by threeseed »

Jgrubb wrote: post #17
beambot wrote: post #13

Looks like a fair number of key aws systems rely on DynamoDB -- and further, the same system used by customers. I wonder: do they have any inclination to decouple these dependencies to prevent correlated outages?

I don't think they rely on Dynamo, they rely on an internal metadata service that Dynamo just happened to overwhelm with too many large requests.

"Amazon SQS uses an internal table stored in DynamoDB to store information describing its queues"

"EC2 Auto Scaling stores information about its groups and launch configurations in an internal table in DynamoDB"

"CloudWatch uses an internal table stored in DynamoDB"

"Customers attempting to log into the Console during this period saw much higher latency in the login process. This was due to a very long timeout being set on an API call that relied on DynamoDB"

Seems like poor architectural design to have all of these storing state in the same instances of DynamoDB that are used by customers. If a new feature like GSI is added it should under no circumstances ever impact other services.

Re: Summary of the Amazon DynamoDB Service Disruption

#22

Post by ckozlowski »

beambot wrote: post #20
Jgrubb wrote: post #17

Earlier quoted context omitted.

I don't think they rely on Dynamo, they rely on an internal metadata service that Dynamo just happened to overwhelm with too many large requests.

From the 2nd sentence: > ... subsequent impact to other AWS services that depend on DynamoDB ... And the metadata service is part of DynamoDB: > The membership of a set of table/partitions within a server is managed by DynamoDB’s internal metadata service.

I just asked the question in a separate post before seeing yours, but I questioned this as well. Would it have been better to have a separate instance of DynamoDB independent from the one customers are writing to? I understand this is added cost/overhead, but it would have resulted in isolating the failure to the customer side only, I'd think.

Re: Summary of the Amazon DynamoDB Service Disruption

#23

Post by eli »

ckozlowski wrote: post #19

I noticed that in the "Impact on other services" bit, that CloudWatch and Console were affected, becasue they were dependent on DyanmoDB. Now, I don't pretend to know DyanmoDB to well, but it seems to me that having your monitoring application dependent on one of the things you would be monitoring is a strange circular dependency. Would it have been wiser for Amazon to implement a completely separate instance of Dyna…

And then what? Set up another set of monitoring tools to watch this new DynamoDB instance that's only used for monitoring and console?

Re: Summary of the Amazon DynamoDB Service Disruption

#25

Post by pfortuny »

bbrazil wrote: post #6
chillydawg wrote: post #5

Earlier quoted context omitted.

How is it any harder to do in the cloud than on a rack in a warehouse? At least you don't have to muck about with cables and phoning power companies up.

Given they had ~300 minutes of outage in 3 years, you're looking at ~99.98% reliable in just that region. That's pretty good for a stateful serving system, and indeed you'd be pushed to do better.

That is just time not traffic loss, it would be hard to get so much traffic lost by a single in-house service.

Metrics depend.

Re: Summary of the Amazon DynamoDB Service Disruption

#26

Post by bboreham »

> We did not have detailed enough monitoring for this dimension (membership size)

It is often the case that you aren't monitoring the metric that would have told you the world was about to end. However,

> their processing time was near the time limit for retrieval

This is a specific response time with a known, fatal, limit. Not seeing that trend seems unfortunate.

Re: Summary of the Amazon DynamoDB Service Disruption

#27

Post by zwily »

ckozlowski wrote: post #19

I noticed that in the "Impact on other services" bit, that CloudWatch and Console were affected, becasue they were dependent on DyanmoDB. Now, I don't pretend to know DyanmoDB to well, but it seems to me that having your monitoring application dependent on one of the things you would be monitoring is a strange circular dependency. Would it have been wiser for Amazon to implement a completely separate instance of Dyna…

It sounds like most of the remediation is "we'll make sure this service keeps working even if Dynamo is down". It's telling that so many services are relying heavily on Dynamo now. If anything, this event just makes me like it more. (I have several things built on it).

Re: Summary of the Amazon DynamoDB Service Disruption

#28

Post by zwily »

deskamess wrote: post #9

> but we should have posted the green-i to the dashboard sooner than we did Use full orange or even half orange circle or something else if you want to convey 'subset of customers may have issues'. If possible move 'having issue' items to the top - this way people do not have to scroll down to see whats broken.

This was the first time we had actually seen red icons on the dashboard. We joked that they had to furiously photoshop them up Sunday morning.

(I'm sure it had happened before, just the first time we had seen them.)

Re: Summary of the Amazon DynamoDB Service Disruption

#29

Post by brudgers »

ghshephard wrote: post #11

A pretty good post mortem, but one worrisome final comment: "For us, availability is the most important feature of DynamoDB" I would think that durability should be the most important feature of DynamoDB; better to have intermittent outage, or reduced capacity, but not have data loss - but perhaps there is something about DynamoDB which suggests availability is rated more highly than durability?

Since the \A\ in ACID is atomicity, I suspect the reading context for "availability" is CAP.

Re: Summary of the Amazon DynamoDB Service Disruption

#30

Post by [deleted] »

ghshephard wrote: post #11

A pretty good post mortem, but one worrisome final comment: "For us, availability is the most important feature of DynamoDB" I would think that durability should be the most important feature of DynamoDB; better to have intermittent outage, or reduced capacity, but not have data loss - but perhaps there is something about DynamoDB which suggests availability is rated more highly than durability?

[deleted]
Post reply on HN