Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon DynamoDB Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

31–40 of 89 posts

Re: Summary of the Amazon DynamoDB Service Disruption

#31

Post by allengeorge »

userbinator wrote: post #2

Unavailable servers continued to retry requests for membership data, maintaining high load on the metadata service. It sounds like the retries exacerbated the situation. I wonder if they use exponential backoff?

Most exponential backoff implementations I've seen have an upper limit, where the initial (fast) retries work around transient networking blips.

My guess is that a case like this you'd hit the backoff's upper bound pretty quickly, and, given the large volume of servers hitting a comparatively small metadata pool, experience exactly the same failure mode.

Re: Summary of the Amazon DynamoDB Service Disruption

#32

Post by _alex_ »

ckozlowski wrote: post #19

I noticed that in the "Impact on other services" bit, that CloudWatch and Console were affected, becasue they were dependent on DyanmoDB. Now, I don't pretend to know DyanmoDB to well, but it seems to me that having your monitoring application dependent on one of the things you would be monitoring is a strange circular dependency. Would it have been wiser for Amazon to implement a completely separate instance of Dyna…

CloudWatch is a monitoring service that AWS offers publicly. It is different from the monitoring service that AWS services use internally.

source: was on an AWS service team for several years

Re: Summary of the Amazon DynamoDB Service Disruption

#33

Post by chillydawg »

antirez wrote: post #8
chillydawg wrote: post #5

Earlier quoted context omitted.

How is it any harder to do in the cloud than on a rack in a warehouse? At least you don't have to muck about with cables and phoning power companies up.

Just an example: during the issue even people serving 10 ops/sec, but very important 10 ops/sec, were affected by a huge complessive load which was not their for most of the part. It's true that when you "go cloud" you don't have to manage your operations, but you are basically putting everything in the hands of other op people, and what happens to you is related to a more wide set of conditions. So managing your stu…

Yeah, but it's a very rare in-house team that can keep stuff up better than an AWS of a gcloud. I'd only consider it if I was doing really, really REALLY specialised stuff that absolutely could not be handed off to some mega-host.

Re: Summary of the Amazon DynamoDB Service Disruption

#37

Post by cddotdotslash »

beambot wrote: post #13

Looks like a fair number of key aws systems rely on DynamoDB -- and further, the same system used by customers. I wonder: do they have any inclination to decouple these dependencies to prevent correlated outages?

A large number of AWS services actually use other AWS services under the hood. Lambda is actually running code on EC2 stored on S3. API Gateway is actually CloudFront on the front end. Apparently most services use DynamoDB for metadata. I'm pretty sure CloudWatch logs are stored on S3. They really interconnect a lot of services. You could argue that it's good (dog fooding, better monitoring) or bad (too many dependencies) but at least they're putting confidence in their own products.

Re: Summary of the Amazon DynamoDB Service Disruption

#38

Post by lostcolony »

djb_hackernews wrote: post #34

Hmm, we are seeing very high request latencies with DynamoDB in multiple accounts and can't even load the DynamoDB console...Anyone else?

I can load the console, but it's very, very slow. And creating a new table takes forever, then spews error text over the interface in a way that makes it clear they never intended for an error to be displayed this way.
Post reply on HN