81–89 of 89 posts
Earlier quoted context omitted.
Green traditionally means we're good, ready to go, etc. Non-intuitive to use any green indicator for a situation deserving a write-up like this. It's yellow at best.
During the actual outage this write up was about, didn't it go red?
Earlier quoted context omitted.
CloudWatch is a monitoring service that AWS offers publicly. It is different from the monitoring service that AWS services use internally. source: was on an AWS service team for several years
It's still a problem even if it's not the internal monitoring system. Customers use CloudWatch to detect problems with DynamoDB and get notified. This dependency means customers may not get notified if CloudWatch does not work as expected due to a DynamoDB problem.
On one hand, one can design a service with minimum dependencies to survive other services' outages, but that means duplication of effort and increased cost, as well as slower delivery of features. Or one can focus on adding value.
It seems though, from the update, that CloudWatch is going to have some sort of caching for most recent data.
Earlier quoted context omitted.
How is it any harder to do in the cloud than on a rack in a warehouse? At least you don't have to muck about with cables and phoning power companies up.
You call up IBM. You ask for a mainframe solution for two sites. You get experts to set it up for you with your application and such. You don't worry about downtime again for at least 30 years. You call up Bull, Fujitsu, or Unisys for the same thing. You call up HP. You ask for a NonStop solution. You get same thing for at least 20 years. You call up VMS Software. You ask for an OpenVMS cluster. You get same thing fo…
One day, they failed. One went offline - for reasons never revealed, at least to me - and the secondary didn't come up. Radio silence, kaput. But an airline that housed mainframes in the same DCs had their booking system fail at exactly the same times (with national headlines to match).
The myth of mainframe uptime is exactly that. La-la-land for hardware & services salesmen.
Earlier quoted context omitted.
How is it any harder to do in the cloud than on a rack in a warehouse? At least you don't have to muck about with cables and phoning power companies up.
You call up IBM. You ask for a mainframe solution for two sites. You get experts to set it up for you with your application and such. You don't worry about downtime again for at least 30 years. You call up Bull, Fujitsu, or Unisys for the same thing. You call up HP. You ask for a NonStop solution. You get same thing for at least 20 years. You call up VMS Software. You ask for an OpenVMS cluster. You get same thing fo…
Earlier quoted context omitted.
The question of timing of the green-I is valid, but different from your original complaint. In your hypothetical scenario, the customer that is going to the panel to "find out what the fuck is going on" is not going to miss the indicator i, and will read the description of the message.
I don't care what that "i" indicator is; if you're showing green while you're impacted, your status board is useless and I'm going to Twitter and HN to see if you're broken (which is what I already have to do for AWS).
Earlier quoted context omitted.
You call up IBM. You ask for a mainframe solution for two sites. You get experts to set it up for you with your application and such. You don't worry about downtime again for at least 30 years. You call up Bull, Fujitsu, or Unisys for the same thing. You call up HP. You ask for a NonStop solution. You get same thing for at least 20 years. You call up VMS Software. You ask for an OpenVMS cluster. You get same thing fo…
This is absolute cobblers. I worked in an IT team that had a pair of IBM mainframes that were fed and watered at crushing expense and for which even the tiniest software change required a colossal waterfall project. One day, they failed. One went offline - for reasons never revealed, at least to me - and the secondary didn't come up. Radio silence, kaput. But an airline that housed mainframes in the same DCs had thei…
Meanwhile, the companies I've worked at all had mainframes without trouble from them that people said. Problems were virtually always the app developers or the pain of doing 21st century stuff with 60's-80's architecture or legacy code.
Earlier quoted context omitted.
You call up IBM. You ask for a mainframe solution for two sites. You get experts to set it up for you with your application and such. You don't worry about downtime again for at least 30 years. You call up Bull, Fujitsu, or Unisys for the same thing. You call up HP. You ask for a NonStop solution. You get same thing for at least 20 years. You call up VMS Software. You ask for an OpenVMS cluster. You get same thing fo…
Caveat here. Sure, a mainframe/high end hardware & supervisor OS can run for 30 years... but the actual applications that users are facing... no, they cannot. You need to upgrade DB2 or IMS or whatever Java app you are using? There will be downtime.
Basic strategy was putting something in front of them that can redirect to the new system upon a trigger. Let's assume its functionality + tons of data. The new system first gets the data moved to it in batch form for efficiency reasons. Once it catches up enough, it starts syncing in a more online fashion until it gets to point that it's syncing in real-time. All kinds of tests are performed on that system throughout this process. Eventually, a change-over happens that should be barely perceptible. The inability to do this is usually due to fragile architecture or tightly-coupled implementations which are unfortunately all too common in enterprises.
Note: It can also help if your app was written in something like Common LISP or Erlang that supports live updates. That with the delta approach (version A->A/B->B) equals upgrades with no downtime. ;) Combining it with clustering approach is quite powerful but clustering approach is more applicable to tools majority uses.
Earlier quoted context omitted.
I think this is written from an ops point of view. Dynamo-based systems allow a client to wait until at least X servers have acknowledged the write, and use eventual consistency in case there is some service disruption. I personally didnt know they used meta data tables in Dynamo, and that sounds like they lost one of the most important traits which gave Dynamo its super-high availability abilities, which is having a…
Thank you for saying that! I just read the Dynamo paper the other day and that was the first thing that jumped out at me when I read this outage description. Now I feel like I understood the paper.
http://www.allthingsdistributed.com/2012/01/amazon-dynamodb....
A pretty good post mortem, but one worrisome final comment: "For us, availability is the most important feature of DynamoDB" I would think that durability should be the most important feature of DynamoDB; better to have intermittent outage, or reduced capacity, but not have data loss - but perhaps there is something about DynamoDB which suggests availability is rated more highly than durability?
If I'm reading their post-outage actions correctly, they certainly align with availability > durability. There are many things to like about DynamoDB, but one of the things I really dislike is the 'abscence of an error == success' pattern that they implement for many operations. With less frequent metadata requests, I imagine there is a greater chance of silent failures (ie availability looks fine, durability is gone…
If you suspect the design of DynamoDB is sloppy, I encourage you to read this: http://cacm.acm.org/magazines/2015/4/184701-how-amazon-web-s...