Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS US East is experiencing high error rates on several services

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 184 posts

Re: AWS US East is experiencing high error rates on several services

#121

Post by justinholmes »

virtuallynathan wrote: post #34
taf2 wrote: post #6

The main issue appears to be DynamoDB Here's a copy from the status page. 3:00 AM PDT We are investigating increased error rates for API requests in the US-EAST-1 Region. 3:26 AM PDT We are continuing to see increased error rates for all API calls in DynamoDB in US-East-1. We are actively working on resolving the issue. 4:05 AM PDT We have identified the source of the issue. We are working on the recovery. 4:41 AM PD…

6:19 AM PDT The metadata service is now stable and we are actively working on removing throttles. 7:12 AM PDT We continue to work on removing throttles and restoring API availability but are proceeding cautiously. 7:22 AM PDT We are continuing to remove throttles and enable traffic progressively. 7:40 AM PDT We continue to remove throttles and are starting to see recovery. 7:50 AM PDT We continue to see recovery of r…

9:12 AM PDT Between 2:13 AM and 8:15 AM PDT we experienced increased error rates for API requests in the US-EAST-1 Region. The issue has been resolved and the service is operating normally.

Re: AWS US East is experiencing high error rates on several services

#123

Post by smoyer »

onetom wrote: post #110

For ADHD I would recommend Concerta/Ritalin/Adderall; that would enable you to read for not just a few seconds more but even minutes more before you judge. (I'm not joking here, I'm using 36mg Concerta for about about 4yrs now. Before that I also often came across as an asshole.) There is also a less evasive way of improving your online communication quality: http://www.paulgraham.com/disagree.html

Are you "Da Tom" in datomic? I'd suggest you be a tad more polite to those who ask honest questions or make statements of opinion here. And if you are "da tom", do you really want to alienate a potential customer or should you take their message to heart? I'm not going to bother looking at your page (and I don't need medication for it), but I'd guess there's no elevator pitch.

Re: AWS US East is experiencing high error rates on several services

#124

Post by TheAceOfHearts »

cwilkes wrote: post #104
fidz wrote: post #63

Earlier quoted context omitted.

From Heroku Status Page: > Our service provider is still working towards resolution of this issue. We will update when we have news, or in 1 hour. I wonder why they don't tell that AWS is their service provider. Is it wrong to make the information less obscure?

Because it is kind of awkward PR for Salesforce: come build on our platform! Oh by the way we haven't even moved Heroku, a company that we bought 4 years ago, over to it yet.

Why move it if it works fine as-is? I think Waze was bought by Google in 2013 and AFAIK, they're also still in AWS.

Re: AWS US East is experiencing high error rates on several services

#126

Post by cbhl »

TheAceOfHearts wrote: post #124
cwilkes wrote: post #104

Earlier quoted context omitted.

Because it is kind of awkward PR for Salesforce: come build on our platform! Oh by the way we haven't even moved Heroku, a company that we bought 4 years ago, over to it yet.

Why move it if it works fine as-is? I think Waze was bought by Google in 2013 and AFAIK, they're also still in AWS.

The PR is still awkward: "come use Google Cloud Services!" while it's not good/easy enough for your acquisitions to migrate to it from AWS.

Re: AWS US East is experiencing high error rates on several services

#127

Post by nzadrozny »

fidz wrote: post #63
colinbartlett wrote: post #7

This is manifesting itself as downtime for a lot of companies, including Heroku: https://status.heroku.com If you want alerts on this sort of thing, my side project StatusGator https://statusgator.io will alert you when services post downtime on their status pages. My dashboard blew up this morning with a ton of red and yellow as soon as Amazon started flaking. Edit: I suppose it's time to invest in a multi-region se…

From Heroku Status Page: > Our service provider is still working towards resolution of this issue. We will update when we have news, or in 1 hour. I wonder why they don't tell that AWS is their service provider. Is it wrong to make the information less obscure?

> I wonder why they don't tell that AWS is their service provider.

It's because Heroku's choice of vendors shouldn't matter to their customers. They see it as an implementation detail, and their responsibility to manage.

So I don't think that's an obfuscation. The people I know at Heroku all have an attitude of, "The buck stops here."

Re: AWS US East is experiencing high error rates on several services

#128

Post by kordless »

cle wrote: post #120
nasalgoat wrote: post #8

This seems like another reason to not rely on Amazon-specific services, other than the obvious vendor lock-in. At least in the event of an instance outage you could conceivably migrate off Amazon to another VPS provider. No one using DynamoDB has an alternative.

How would this have helped? In the event of an AWS outage, you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure. This outage lasted a few hours, wouldn't it take you that long to switch providers anyway? If you don't abdicate responsibility for maintaining the infrastructure, you lose a lot of the operational benefits of using a cloud service...in general, I think it…

> you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure

No, they are fixing their infrastructure. The point here is that all single provider systems are destined for periodic failure. Not relying on one single provider is, in theory, a service's means of providing higher reliability. This is the general argument for regions and availability zones on Amazon, but that relies on trusting there is no single point of failure with the system (i.e. inside Amazon).

I've worked for several companies who run the majority of their services on AWS, yet maintain functional systems on other providers in the case of an AWS outage.

Re: AWS US East is experiencing high error rates on several services

#130

Post by kordless »

divideby0 wrote: post #30
drendorx39 wrote: post #11

Earlier quoted context omitted.

failure tolerance is an alien technology for amazon...

that's completely untrue. there are many ways to do fault-tolerance in AWS. it's expensive, but it's possible. netflix even goes as far as simulating the failure of entire aws regions in their simian army testing suite: http://techblog.netflix.com/2011/07/netflix-simian-army.html That's why Netflix stays up when us-east or us-west are down.

It is true. Amazon offloads a decent amount of fault tolerance to the application provider, as you point out here. I will also mention that Netflix does not solely rely on Amazon for running their services. They run their own decentralized caching layer: https://openconnect.netflix.com/deliveryOptions/
Post reply on HN