Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS having major issues

status.aws.amazon.com

Post reply on HN
Print view View original

41–50 of 69 posts

Re: AWS having major issues

#41

Post by javiercr »

We have 4 projects in Amazon OpsWorks, all of them in eu-west. Only one of them has been affected. ELB removed instances from balancing, and then instances were automatically shut down. Now when we try to create new instances they freeze with "requested" status.

What an awful way to start a day (in Europe).

Update: we just found another of our projects has the worker instance being shut down (by Amazon, we didn't touch it).

Update 2: our new instances are finally being created and running setup.

Update 3: we found that the Health Check for our Elastic Load Balancer changed its path to the default one (index.html). We had to edit it again.. This is weird, I don't like the idea of Amazon changing our Health Check path for no reason.

Re: AWS having major issues

#43

Post by twykke »

all our opsworks servers have been unreachable from the internet since around 05:13Z when we see a mysterious 'configure' command with no further details in the opsworks logs

EDIT: they are in EU-West zone btw

Re: AWS having major issues

#44

Post by K0nserv »

javiercr wrote: post #41

We have 4 projects in Amazon OpsWorks, all of them in eu-west. Only one of them has been affected. ELB removed instances from balancing, and then instances were automatically shut down. Now when we try to create new instances they freeze with "requested" status. What an awful way to start a day (in Europe). Update: we just found another of our projects has the worker instance being shut down (by Amazon, we didn't tou…

Most of our OpsWorks projects in EU-West went down and all the critical ones at that. I envy your luck

Re: AWS having major issues

#46

Post by ghayes »

javiercr wrote: post #41

We have 4 projects in Amazon OpsWorks, all of them in eu-west. Only one of them has been affected. ELB removed instances from balancing, and then instances were automatically shut down. Now when we try to create new instances they freeze with "requested" status. What an awful way to start a day (in Europe). Update: we just found another of our projects has the worker instance being shut down (by Amazon, we didn't tou…

I'm in the same situation on US West on OpsWorks. Servers all began to terminate and will not start back up. I put too many eggs in the OpsWorks basket, I suppose.

Re: AWS having major issues

#47

Post by codingninja »

Fortunately the Opsworks service wasn't able to contact EC2 API so although it attempted to terminate the servers, the termination call failed so the instances are still alive and functioning but there is now a disconnect between Opsworks and EC2

Re: AWS having major issues

#48

Post by MichaelCrawford »

beilabs wrote: post #29

So three different environments have had the ELB just release all of their clients and not come back onboard. Manual intervention was the only way. Whats the recommended way to monitor if instances have become detached or non responsive? Want to immediately alert slack or send an email, etc.

djbdns enables one to configure multiple IPs for just one http server.

Host with multiple, geographically distant data centers. If any one server drops offline, remove it from your tinydns "data" file. For that to yield joy you need a shorter timeout than is normally used for webserver DNS.

Each data center must be operated by a different company. The problem tonight appears to be internal to Amazon, so different AWS data centers won't help you.

I only recommend four. The people that run them know more about hosting than you or I could ever hope to:

    http://prgmr.com/

    http://www.budgetdedicated.com/

    http://prq.se/?intl=1
Ivory Tower Group LLC is no longer in the hosting biz but I expect I can convince them to return to it, given tonight's market opportunity. If so I recommend them highly as well, and will post their hosting service URL in a reply.

PRQ is somewhat pricy, but:

       Our boundless commitment to free speech has been 
       tested and proven over and over again. If it is
       legal in Sweden, we will host it, and will keep it 
       up regardless of any pressure to take it down.
       We have ZERO tolerance against SPAM and related services!

       We defend your integrity to the end. With our 
       discreet customer relations policy we don't even 
       have to know who you are, and if we do, we will keep 
       that knowledge strictly confidential.
PRQ has its own in-house legal team to fight takedown demands. However they only rent physical servers, they don't do VPS.

Truth In Labeling: BudgetDedicated and prgmr.com host me for free, prgmr.com because I helped them with their marketing, BudgetDedicated because the server I use hosts an Open Source and a Free Software project.

Re: AWS having major issues

#49

Post by chris7 »

javiercr wrote: post #41

We have 4 projects in Amazon OpsWorks, all of them in eu-west. Only one of them has been affected. ELB removed instances from balancing, and then instances were automatically shut down. Now when we try to create new instances they freeze with "requested" status. What an awful way to start a day (in Europe). Update: we just found another of our projects has the worker instance being shut down (by Amazon, we didn't tou…

All our production servers in OpsWorks had "Connection lost" status. Currently most of them is back "online" but from time time downtime (few minutes) comes back on some instances. AWS Support is not working (50x and 40x errors) - they have moved the support for production to amazon.com
Post reply on HN