71–80 of 184 posts
We were in the middle of a large infrastructure change starting at 4:30am this morning, including taking our application offline. I'm very thankful that we did dry runs along with timing how long certain operations like RDS restores should take and planned for abort steps in case something goes wrong. We noticed that RDS and ElastiCache backup and restores were taking much longer than expected, and once the first set…
This is manifesting itself as downtime for a lot of companies, including Heroku: https://status.heroku.com If you want alerts on this sort of thing, my side project StatusGator https://statusgator.io will alert you when services post downtime on their status pages. My dashboard blew up this morning with a ton of red and yellow as soon as Amazon started flaking. Edit: I suppose it's time to invest in a multi-region se…
From Heroku Status Page: > Our service provider is still working towards resolution of this issue. We will update when we have news, or in 1 hour. I wonder why they don't tell that AWS is their service provider. Is it wrong to make the information less obscure?
Earlier quoted context omitted.
I don't disagree that you would want to have a rough idea of what migrating off of DynamoDB would require but wouldn't the easier step be using redundancy across regions first? Most of the sites which have suffered downtime due to AWS outages have been operating in only a single region (or even AZ!) and adding the extra level isolation is usually going to be a lot easier than dealing with multiple vendors or having t…
In this case AWS sign in is down (perhaps it depends on Dynamo), so redundancy across regions can only help you if fail over is fully automatic which has it's own problems (split brain syndrome...)
Earlier quoted context omitted.
I believe he meant things like using RabbitMQ instead of SQS (SQS isn't very good anyway IMHO) and things like that. More services running on plain old instances rather than AWS specific APIs, Azure specific APIs, etc. This prevents you from getting too locked in and losing control over your application. I honestly can't believe that Netflix can't even load their home page without DynamoDB and all this other stuff. E…
Yes, exactly. It's tempting to drink the Kool-Aid on the various services but the more you rely on them, the less flexible you become and more open to overall service failures like this one. You can take advantage of the benefits of EC2 without relying on the other services. I mean, if even Netflix can't stay up during this, what hope does a startup have?
Earlier quoted context omitted.
DynamoDB now support cross-region replication [0] so you can build more resilient applications with it [0] http://docs.aws.amazon.com/amazondynamodb/latest/developergu...
I don't think cross-region replication would've helped in this case: "The replica tables are intended to serve as read-only copies of the data; however, it is possible to write data to a replica table. If you write data to a replica, those changes will not be propagated to the master, or to any other replicas."
Earlier quoted context omitted.
failure tolerance is an alien technology for amazon...
that's completely untrue. there are many ways to do fault-tolerance in AWS. it's expensive, but it's possible. netflix even goes as far as simulating the failure of entire aws regions in their simian army testing suite: http://techblog.netflix.com/2011/07/netflix-simian-army.html That's why Netflix stays up when us-east or us-west are down.
The AWS KMS is not working. Critital payment applicaction down =S.
Why is us-east-1 so terrible? All of the downtime this year has been Virginia.