AWS having major issues • Hacker News Board

Re: AWS having major issues

#51

Post by toriaezu » Fri, Jul 31, 2015, 8:14 AM UTC

javiercr wrote: post #41

We have 4 projects in Amazon OpsWorks, all of them in eu-west. Only one of them has been affected. ELB removed instances from balancing, and then instances were automatically shut down. Now when we try to create new instances they freeze with "requested" status. What an awful way to start a day (in Europe). Update: we just found another of our projects has the worker instance being shut down (by Amazon, we didn't tou…

Hey javiecr,

Can you please post your OpsWorks instances ids on the AWS forums and just c/p what you wrote here?

https://forums.aws.amazon.com/forum.jspa?forumID=153

thanks.

Re: AWS having major issues

#52

Post by javiercr » Fri, Jul 31, 2015, 8:17 AM UTC

toriaezu wrote: post #51

javiercr wrote: post #41

We have 4 projects in Amazon OpsWorks, all of them in eu-west. Only one of them has been affected. ELB removed instances from balancing, and then instances were automatically shut down. Now when we try to create new instances they freeze with "requested" status. What an awful way to start a day (in Europe). Update: we just found another of our projects has the worker instance being shut down (by Amazon, we didn't tou…

Hey javiecr, Can you please post your OpsWorks instances ids on the AWS forums and just c/p what you wrote here? https://forums.aws.amazon.com/forum.jspa?forumID=153 thanks.

Do you work for Amazon? We already sent a support ticket using http://www.amazon.com/gp/html-forms-controller/support-cente...

Re: AWS having major issues

#53

Post by j42 » Fri, Jul 31, 2015, 9:04 AM UTC

Chiming in as I was completely unaffected... maybe this can help someone here.

I switched our applications off of OpsWorks a couple months back after losing faith in the OW team, in favor of a lower-level CI flow using auto-scaling groups and CodeDeploy.

Aside from some headaches in the very beginning (where CodeDeploy maintenance window would crash the daemon and health checks would cause an "initialize/destruct" loop) it precluded this issue entirely.

Two of our boxes were affected by the outage and disabled, however they were re-initialized within minutes by the ASG, meaning we experienced essentially no downtime.

Just be aware that if you use CodeDeploy, it's essentially just a low-level deployment hook which takes the packaged revision passed to S3 from your continuous integration setup, unpackages and runs any initialization scripts you require. You'll need to configure the security groups and scaling policies on your own which is something I know OpsWorks tries to make easier with their higher level app/layer constructs...

Re: AWS having major issues

#54

Post by vaibhavrajput » Fri, Jul 31, 2015, 9:18 AM UTC

AWS SES 454 Temporary service failure

Re: AWS having major issues

#55

Post by halotrope » Fri, Jul 31, 2015, 9:35 AM UTC

We where also affected in Frankfurt, really scary. Does anyone know about some managed chef like Ops-Work that would be cloud-agnostic with a little less vendor-lock in?

Re: AWS having major issues

#56

Post by fizx » Fri, Jul 31, 2015, 9:38 AM UTC

Their status history is 100% green for the last week. You must be wrong about major issues. /s

Re: AWS having major issues

#57

Post by systemz » Fri, Jul 31, 2015, 9:46 AM UTC

There is your reliable cloud :) You can always count on cloud, right?

Re: AWS having major issues

#58

Post by vacri » Fri, Jul 31, 2015, 9:56 AM UTC

kacy wrote: post #21

Ok. If you're receiving errors, and you're NOT using opsworks, please respond. We're using opsworks too and have ~30 servers down. Maybe we all should be looking at the opsworks agent.

I lost my buildserver - uncontactable, but not terminated. Can't even 'force off'. It had nothing to do with any form of AWS provisioning (manual ansible; a job for Monday), and it's a relatively recent machine (couple of months, t2.medium). Got an email from AWS that the host had degraded, and I noticed the instance was having weird disk issues earlier.

"Description: The instance is running on degraded hardware"

Re: AWS having major issues

#59

Post by vacri » Fri, Jul 31, 2015, 10:04 AM UTC

Rapzid wrote: post #32

beilabs wrote: post #29

So three different environments have had the ELB just release all of their clients and not come back onboard. Manual intervention was the only way. Whats the recommended way to monitor if instances have become detached or non responsive? Want to immediately alert slack or send an email, etc.

There are a lot of ways to integrate, but we use datadog and I generally find it excellent. Alerts to hipchat, pagerduty, etc, with AWS Cloudwatch integration. Of course, you have to assume the Cloudwatch API is working... :)

A vote for datadog here too. Just started using it - it's clear that someone there really loves data.

Re: AWS having major issues

#60

Post by fredonrails » Fri, Jul 31, 2015, 10:04 AM UTC

j42 wrote: post #53

Chiming in as I was completely unaffected... maybe this can help someone here. I switched our applications off of OpsWorks a couple months back after losing faith in the OW team, in favor of a lower-level CI flow using auto-scaling groups and CodeDeploy. Aside from some headaches in the very beginning (where CodeDeploy maintenance window would crash the daemon and health checks would cause an "initialize/destruct" lo…

Thanks for sharing your ideas. Actually I want to do that as well. I'm so tired of Opsworks, failing deploy and of course today issue.

Could you share more about your code-deploy setup and auto-scaling ?