Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon DynamoDB Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

61–70 of 89 posts

Re: Summary of the Amazon DynamoDB Service Disruption

#61

Post by toomuchtodo »

deanCommie wrote: post #55
felixgallo wrote: post #50

Earlier quoted context omitted.

different severity levels are fine. That's what the indicator is for. Decorating the indicator, normally completely incorrectly, with a tiny extra indicator, is incomprehensible. That's why you've never seen a tiny red light in the upper corner of your green traffic light. Your 'check engine' light is not green with a tiny extra indicator in the corner. The metaphors go on and on. Customers are totally not interested…

The question of timing of the green-I is valid, but different from your original complaint. In your hypothetical scenario, the customer that is going to the panel to "find out what the fuck is going on" is not going to miss the indicator i, and will read the description of the message.

I don't care what that "i" indicator is; if you're showing green while you're impacted, your status board is useless and I'm going to Twitter and HN to see if you're broken (which is what I already have to do for AWS).

Re: Summary of the Amazon DynamoDB Service Disruption

#62

Post by toomuchtodo »

colinbartlett wrote: post #43
yandie wrote: post #41

As this post is on, Dynamo is experiencing error again: 6:33 AM PDT We are investigating increased error rates for API requests in the US-EAST-1 Region. Source: http://status.aws.amazon.com/

Some services are reporting problems now. Starting with the same set as last time such as Heroku and lots of services that depend on it: https://statusgator.io/events

DynamoDB, Cloudwatch, EC2, Scaling, and Elastic Beanstalk are all impacted currently.

Re: Summary of the Amazon DynamoDB Service Disruption

#63

Post by toomuchtodo »

chillydawg wrote: post #5
arturhoo wrote: post #3

We did not have detailed enough monitoring for this dimension (membership size), and didn’t have enough capacity allocated to the metadata service to handle these much heavier requests. As much as I admire and rely on AWS' scale to build architectures and fault tolerant applications, it can't be ignored that the marketing towards going "full cloud" doesn't take into account how hard it is to build resilient architect…

How is it any harder to do in the cloud than on a rack in a warehouse? At least you don't have to muck about with cables and phoning power companies up.

I used to do infrastructure on physical hardware, and we'd go years without an outage sometimes (generators in the datacenter, diesel fuel contracts, redundant fiber providers using BGP). Doing it in the cloud is harder, because you're at the mercy of the provider when things go south, and you have no transparency into why it went wrong except what they're willing to publish. Why did it happen? Will it happen again?

I mean, you can argue that the cloud is better. But how often is Heroku and AWS down? About the same as physical providers (I concede S3 is pretty solid though).

Re: Summary of the Amazon DynamoDB Service Disruption

#64

Post by deskamess »

RyanGWU82 wrote: post #60
deanCommie wrote: post #55

Earlier quoted context omitted.

The question of timing of the green-I is valid, but different from your original complaint. In your hypothetical scenario, the customer that is going to the panel to "find out what the fuck is going on" is not going to miss the indicator i, and will read the description of the message.

It's really easy to miss the indicator "i". I've been doing this for 4 years and it still doesn't catch my eye when scanning the page.

I agree 100%. That column of uniform green makes it easy to miss. It is a poor UI choice but can be easily fixed.

Re: Summary of the Amazon DynamoDB Service Disruption

#65

Post by ckozlowski »

_alex_ wrote: post #32
ckozlowski wrote: post #19

I noticed that in the "Impact on other services" bit, that CloudWatch and Console were affected, becasue they were dependent on DyanmoDB. Now, I don't pretend to know DyanmoDB to well, but it seems to me that having your monitoring application dependent on one of the things you would be monitoring is a strange circular dependency. Would it have been wiser for Amazon to implement a completely separate instance of Dyna…

CloudWatch is a monitoring service that AWS offers publicly. It is different from the monitoring service that AWS services use internally. source: was on an AWS service team for several years

I follow. In this case though, I was thinking of the customer's monitoring tools. Their first action would be to try a diagnose themselves.

But good to know! Thanks!

Re: Summary of the Amazon DynamoDB Service Disruption

#66

Post by reustle »

Someone1234 wrote: post #56
yandie wrote: post #41

As this post is on, Dynamo is experiencing error again: 6:33 AM PDT We are investigating increased error rates for API requests in the US-EAST-1 Region. Source: http://status.aws.amazon.com/

Their status screen is totally misleading and utterly pointless. Someone needs to get all of the RSS feeds (which are actually accurate) and create a new dash which is honest.

Are the RSS feeds more accurate than the status page?

Re: Summary of the Amazon DynamoDB Service Disruption

#67

Post by Someone1234 »

reustle wrote: post #66
Someone1234 wrote: post #56

Earlier quoted context omitted.

Their status screen is totally misleading and utterly pointless. Someone needs to get all of the RSS feeds (which are actually accurate) and create a new dash which is honest.

Are the RSS feeds more accurate than the status page?

I'd argue that they are.

Re: Summary of the Amazon DynamoDB Service Disruption

#68

Post by sillypog »

stingraycharles wrote: post #15
ghshephard wrote: post #11

A pretty good post mortem, but one worrisome final comment: "For us, availability is the most important feature of DynamoDB" I would think that durability should be the most important feature of DynamoDB; better to have intermittent outage, or reduced capacity, but not have data loss - but perhaps there is something about DynamoDB which suggests availability is rated more highly than durability?

I think this is written from an ops point of view. Dynamo-based systems allow a client to wait until at least X servers have acknowledged the write, and use eventual consistency in case there is some service disruption. I personally didnt know they used meta data tables in Dynamo, and that sounds like they lost one of the most important traits which gave Dynamo its super-high availability abilities, which is having a…

Thank you for saying that! I just read the Dynamo paper the other day and that was the first thing that jumped out at me when I read this outage description. Now I feel like I understood the paper.

Re: Summary of the Amazon DynamoDB Service Disruption

#69

Post by astral303 »

"We did not realize soon enough that this low overall error rate was giving some customers disproportionately high error rates."

Critical lesson for running multitenant SaaS services at scale: you need to monitor not just overall error rates, but individual tenant/customer error rates.

Re: Summary of the Amazon DynamoDB Service Disruption

#70

Post by nickpsecurity »

deanCommie wrote: post #48
felixgallo wrote: post #47

"It was impacting a relatively small number of customers, but we should have posted the green-i to the dashboard sooner than we did on Monday." Amazon. No. Please listen. You should never post a green-i. Green-i means nothing to anyone. It's a minimization of a problem. You should change the indicator to show that there is a problem. If there is a problem, that is what the indicator is for. Customers don't care about…

What's with this all-or-nothing attitude? What is so wrong about different severity levels? Green-i: Some small percentage of customers are affected, but most of you have nothing to worry about. So if you are a customer having problems and see this notice, you are re-assured that you are not crazy. And if you are NOT seeing problems, you probably won't. Amazon has not always been the best at posting this information…

Green traditionally means we're good, ready to go, etc. Non-intuitive to use any green indicator for a situation deserving a write-up like this. It's yellow at best.
Post reply on HN