Live data from Hacker News

AWS US East is experiencing high error rates on several services

news.ycombinator.com

151–160 of 184 posts

Re: AWS US East is experiencing high error rates on several services

#151

DynamoDB is literally a garbage. That's why Amazon does not provide any SLA for the service... Even cheap Azure Storage provides cross-region failover.

I have a feeling you have no clue what you are talking about, nor read dynamo db paper or tried to use it in a production environment.

Re: AWS US East is experiencing high error rates on several services

#152
The aws services stack is deep and deeply intertwined. I've always viewed depending on such stacks in production with skepticism and I'd recommend everybody else does that too.

This might come across as tooting our horn a bit. But it's more about sounding a warning to other startups providing SaaS service built on public cloud. My own misgivings about relying on a cloud provider specific stack (both for the reasons of visibility/debuggability as well as for vendor lock-in) meant that PacketZoom services were not affected by this failure at all because we only use them as one of the many providers of raw machines. We use our own techniques to load-balance/failover among multiple cloud providers too (so even if the raw compute/network went away, our service would take a perf hit but not be completely down).

Re: AWS US East is experiencing high error rates on several services

#153
post #33

Earlier quoted context omitted.

I don't disagree that you would want to have a rough idea of what migrating off of DynamoDB would require but wouldn't the easier step be using redundancy across regions first? Most of the sites which have suffered downtime due to AWS outages have been operating in only a single region (or even AZ!) and adding the extra level isolation is usually going to be a lot easier than dealing with multiple vendors or having t…

In this case AWS sign in is down (perhaps it depends on Dynamo), so redundancy across regions can only help you if fail over is fully automatic which has it's own problems (split brain syndrome...)

One of the first things Amazon tell you about AWS is the API is king - the console is hosted in us-east-1 but the API is hosted in each region independently.

You don't need automatic failover, but you do need your own scripts to talk to the AWS API. The whole point of AWS is automation.

Re: AWS US East is experiencing high error rates on several services

#154
post #20

Earlier quoted context omitted.

Garbage you say? Dynamodb blazed a trail for many open source, eventually consistent kv databases. Certainly not garbage.

The only thing DynamoDB can do good is simplicity. Except for this, even MongoDB has tons more features than DynamoDB and the new version resolved performance problems existed in the previous versions.

MongoDB really? You are comparing a software product to a multi-region global data service? This is just not a great comparison. If you build a distributed global data service on MongoDB you could compare that to DynamoDB.

Re: AWS US East is experiencing high error rates on several services

#155
You know this is interesting. There were no symptoms for us at all that something was wrong with Amazon itself, and their status page was not updated in a timely fashion. I spent a few hours (in the middle of the night working with my laptop in bed next to my wife) trying to figure out what in the hell was wrong, only to find out through the grapevine that it was Amazon. This is extremely frustrating when providers are having problems and actively working on a solution yet their status page still has glowing recommendations of their service.

Re: AWS US East is experiencing high error rates on several services

#156

The aws services stack is deep and deeply intertwined. I've always viewed depending on such stacks in production with skepticism and I'd recommend everybody else does that too. This might come across as tooting our horn a bit. But it's more about sounding a warning to other startups providing SaaS service built on public cloud. My own misgivings about relying on a cloud provider specific stack (both for the reasons o…

Or you could just run in multiple regions. Using multiple cloud providers limits your ability to take advantage of provider-specific features - why waste time writing your own load balancer when you could use ELB + multiple regions?

Re: AWS US East is experiencing high error rates on several services

#157

This seems like another reason to not rely on Amazon-specific services, other than the obvious vendor lock-in. At least in the event of an instance outage you could conceivably migrate off Amazon to another VPS provider. No one using DynamoDB has an alternative.

DynamoDb was only down in only one region, for the first time in years. It's hardly a reason to migrate off, especially with DynamoDb still being available in every single other region throughout. It's simpler to fail over to another AWS region than to an entirely different cloud provider with a different API.

Re: AWS US East is experiencing high error rates on several services

#158

The aws services stack is deep and deeply intertwined. I've always viewed depending on such stacks in production with skepticism and I'd recommend everybody else does that too. This might come across as tooting our horn a bit. But it's more about sounding a warning to other startups providing SaaS service built on public cloud. My own misgivings about relying on a cloud provider specific stack (both for the reasons o…

Or you could just run in multiple regions. Using multiple cloud providers limits your ability to take advantage of provider-specific features - why waste time writing your own load balancer when you could use ELB + multiple regions?

"Or you could just run in multiple regions."

Not when the original goal of the very service is to have presence in all geographical regions. If aws us-east is hit, I want the users to transparently failover to a server on east coast (perhaps one hosted by google or softlayer) rather than be directed all the way to us-west or eu.

And as for ELB, one doesn't use ELB for a custom protocol that load-balances/fails-over itself from the client :-)

Re: AWS US East is experiencing high error rates on several services

#159

What is happening to stock price? oh; its sunday; forgot we can't trade.

I don't recall any outages having a material effect on Amazon's stock price.

Amazon can apparently go up or down 50% depending on whether a butterfly sneezes in Australia (it's wildly danced between 284 and 580 in the past 12 months alone).

In the background of such high volatility, it would be hard to pinpoint such a material effect from such a small disruption (in the big picture of course, I'm betting there are some pretty angry customers today due to the loss of a few sigma of reliability from this outage alone).

Now if a study were published indicating customers were switching providers over incidents this, then I think you'd have some material evidence. But is anyone else better yet? Azure was out for 12 hours last year apparently...

http://www.datacenterknowledge.com/archives/2015/01/23/cloud...

Re: AWS US East is experiencing high error rates on several services

#160
post #58

Earlier quoted context omitted.

I don't much use it anymore but this is something I always liked about AppEngine: baked into the pricing is automatic replication over three regions. Serious AWS web apps are also distributed across multiple regions/availability zones, but many companies don't do that. It would be good for hosting services like Heroku, which I use, would replicate across zones also. To be fare to Amazon, they tell you you architect a…

dynamo db is affected in the entire region, however, not just a single AZ.

That's why parent post mentioned multiple regions. No other AWS region was affected...
Post reply on HN