Live data from Hacker News

AWS US East is experiencing high error rates on several services

news.ycombinator.com

161–170 of 184 posts

Re: AWS US East is experiencing high error rates on several services

#161
post #63

Earlier quoted context omitted.

From Heroku Status Page: > Our service provider is still working towards resolution of this issue. We will update when we have news, or in 1 hour. I wonder why they don't tell that AWS is their service provider. Is it wrong to make the information less obscure?

> I wonder why they don't tell that AWS is their service provider. It's because Heroku's choice of vendors shouldn't matter to their customers. They see it as an implementation detail, and their responsibility to manage. So I don't think that's an obfuscation. The people I know at Heroku all have an attitude of, "The buck stops here."

Not that i think there's anything wrong with heroku's messaging here, but "we're down because our service provider is having issues" isn't really a "the buck stops here" sort of update.

Re: AWS US East is experiencing high error rates on several services

#162
post #137

Earlier quoted context omitted.

Because it is kind of awkward PR for Salesforce: come build on our platform! Oh by the way we haven't even moved Heroku, a company that we bought 4 years ago, over to it yet.

They will never be able to move Heroku of AWS. Their customers depend on being on the same cloud as other service providers

The location of instances could be an option. I imagine if their locally-hosted instances were a bit cheaper they would be the more popular option.

Re: AWS US East is experiencing high error rates on several services

#163
post #110

For ADHD I would recommend Concerta/Ritalin/Adderall; that would enable you to read for not just a few seconds more but even minutes more before you judge. (I'm not joking here, I'm using 36mg Concerta for about about 4yrs now. Before that I also often came across as an asshole.) There is also a less evasive way of improving your online communication quality: http://www.paulgraham.com/disagree.html

Prescribing psychiatric medication to another user is maybe not the best way to encourage them to comment more thoughtfully.

We detached this comment from https://news.ycombinator.com/item?id=10247727 and marked it off-topic.

Re: AWS US East is experiencing high error rates on several services

#165

Earlier quoted context omitted.

Software bug caused downtime vs infrastructure / hardware availability uptime to me are a different guarantee. I am pretty sure someone did something recently to DynamoDB.

Infrastructure guy here doing this for 14 years. Downtime is downtime. You get a pass if its "scheduled maintenance" you've notified your customers about to allow them to be prepared, but if you silently perform maintenance and it goes to shit, you've just counted against your metrics.

Nope. I still disagree. No service can guarantee 99.999999% unless you discount software upgrade. You just cannot. If you think those nines include software upgrades, you are probably over optimistic.

Re: AWS US East is experiencing high error rates on several services

#166

Earlier quoted context omitted.

Infrastructure guy here doing this for 14 years. Downtime is downtime. You get a pass if its "scheduled maintenance" you've notified your customers about to allow them to be prepared, but if you silently perform maintenance and it goes to shit, you've just counted against your metrics.

Nope. I still disagree. No service can guarantee 99.999999% unless you discount software upgrade. You just cannot. If you think those nines include software upgrades, you are probably over optimistic.

> No service can guarantee 99.999999%

Don't advertise it if you can't offer it then.

> If you think those nines include software upgrades, you are probably over optimistic.

If you advertise a product with a specific SLA, and you can't meet that SLA, you're a liar. Don't try to blame the victim because of inaccurate/untruthful marketing or engineering.

Re: AWS US East is experiencing high error rates on several services

#167

As an AWS customer you need to be aware that the service health of all AWS services and not just the ones you use directly are important. You say you don't use SQS or SNS? When they go down, you might not be able to get Logs or even login to the web Console. Same goes for things like AutoScaling, OpsWorks, etc.

That's the beauty of micro-service architectures. You don't have a single monolithic point of failure, you have dozens of smaller ones.

Re: AWS US East is experiencing high error rates on several services

#169

Earlier quoted context omitted.

Nope. I still disagree. No service can guarantee 99.999999% unless you discount software upgrade. You just cannot. If you think those nines include software upgrades, you are probably over optimistic.

> No service can guarantee 99.999999% Don't advertise it if you can't offer it then. > If you think those nines include software upgrades, you are probably over optimistic. If you advertise a product with a specific SLA, and you can't meet that SLA, you're a liar. Don't try to blame the victim because of inaccurate/untruthful marketing or engineering.

Maybe you are the one who needs to understand their SLA and fine print.

Re: AWS US East is experiencing high error rates on several services

#170
The last time there were API outages in AWS, our autoscaling logic could not determine the number of running instances, so it felt it had too few. It kept launching instances, and due to the API outage we couldn't manually kill the instances either...

So we wound up with over 1,000 of these machines running which then due to our fan out of their DB they needed to load into memory from other machines, our whole environment crashed until we could kill off the erroneously launched instances.

This meant an effective full reboot of our entire platform...

It's was not a fun weekend.

Post reply on HN