1–10 of 184 posts
Dynamo DB is throttling API access and seems to be having issues with the management of meta data.
Here's a copy from the status page.
3:00 AM PDT We are investigating increased error rates for API requests in the US-EAST-1 Region.
3:26 AM PDT We are continuing to see increased error rates for all API calls in DynamoDB in US-East-1. We are actively working on resolving the issue.
4:05 AM PDT We have identified the source of the issue. We are working on the recovery.
4:41 AM PDT We continue to work towards recovery of the issue causing increased error rates for the DynamoDB APIs in the US-EAST-1 Region.
4:52 AM PDT We want to give you more information about what is happening. The root cause began with a portion of our metadata service within DynamoDB. This is an internal sub-service which manages table and partition information. Our recovery efforts are now focused on restoring metadata operations. We will be throttling APIs as we work on recovery.
5:22 AM PDT We can confirm that we have now throttled APIs as we continue to work on recovery.
5:42 AM PDT We are seeing increasing stability in the metadata service and continue to work towards a point where we can begin removing throttles.
If you want alerts on this sort of thing, my side project StatusGator https://statusgator.io will alert you when services post downtime on their status pages. My dashboard blew up this morning with a ton of red and yellow as soon as Amazon started flaking.
Edit: I suppose it's time to invest in a multi-region setup. Since StatusGator is hosted on Heroku in the US-East region, it is in theory affected by this problem though so far is still up.
At least in the event of an instance outage you could conceivably migrate off Amazon to another VPS provider. No one using DynamoDB has an alternative.
Not funny when its Sunday.