Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linode / Akamai US-EAST is down

status.linode.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Linode / Akamai US-EAST is down

#21

Post by ajdude »

lgeek wrote: post #20

Over 22 hours of downtime for the one VPS I have in that region. My infrastructure is redundant and spread out among hosting providers and DCs so there's no real impact, but I'm pretty sure this is the longest outage I've ever had with any provider. And the communication level has been so dissapointing. 4 hrs to say it's a power / HVAC issue? Updates that basically just say we're still working on it since then.

We are approaching 24 hours of downtime, I'm still one of those also affected and I'm starting to wonder if the situation is worse than they are letting on.

Re: Linode / Akamai US-EAST is down

#23

Post by lgeek »

lgeek wrote: post #20

Over 22 hours of downtime for the one VPS I have in that region. My infrastructure is redundant and spread out among hosting providers and DCs so there's no real impact, but I'm pretty sure this is the longest outage I've ever had with any provider. And the communication level has been so dissapointing. 4 hrs to say it's a power / HVAC issue? Updates that basically just say we're still working on it since then.

Update: it's finally up again after around 25.5 hrs of downtime

Re: Linode / Akamai US-EAST is down

#24

Post by ajdude »

lgeek wrote: post #23
lgeek wrote: post #20

Over 22 hours of downtime for the one VPS I have in that region. My infrastructure is redundant and spread out among hosting providers and DCs so there's no real impact, but I'm pretty sure this is the longest outage I've ever had with any provider. And the communication level has been so dissapointing. 4 hrs to say it's a power / HVAC issue? Updates that basically just say we're still working on it since then.

Update: it's finally up again after around 25.5 hrs of downtime

Here's to hoping mine comes up again soon

Re: Linode / Akamai US-EAST is down

#26

Post by cmg »

Of 60-something Linodes in Newark across a few accounts (we don't use LKE, Node Balancers, etc)

- Many came back up yesterday. Most of the rest came back up this morning.

- All but two are back online. One of those is "Powered off" but can't be turned on because "Linode busy". The other is online but unreachable, same behavior as most of them during the outage.

- Three required me to put them in Rescue Mode and run fsck.ext4 -F /dev/sda to get them back online.

Re: Linode / Akamai US-EAST is down

#27

Post by basilgohar »

basilgohar wrote: post #7
dreeves wrote: post #2

Is this hard downtime for anyone else? We're not sure what they mean by "emerging service issue" and they haven't replied to our support ticket yet.

Yes, my servers and all associate services are down: DNS, email, websites. It's a major outage at the whole Newark datacenter, which is their main one, no less.

Thankfully all my nodes are back up. But the DNS server as down so long that Namecheap registered it as a personal DNS server, so I am working with them to get it back up.

These are secondary effects of outage, not Linode directly, but caused by the outage itself.

Re: Linode / Akamai US-EAST is down

#28

Post by basilgohar »

Thankfully all my nodes are back online and accessible via SSH but there are currently secondary effects due to the length of the outage that I am dealing with, such as my DNS server being deregistered with Namecheap from serving as a personal DNS server (their name for that feature). The consequence of this is also that my email is not currently work, since that affects the hostname the email server operates under serving multiple other domains. I am still grateful that it seems I suffered no data loss, thank God!

Also, I've taken it upon myself to PROPERLY implement a better, redundant backup strategy (since I was mainly relying on Linode's service, but now I feel I should go beyond that). I am using restic backing-up to a Backblaze bucket via the S3 interface. Nice thing is I can put all hosts into the same bucket and restic will organize by host but still full deduplication support. Not sure how much that'll net me but it's nice to have.

Re: Linode / Akamai US-EAST is down

#29

Post by ajdude »

basilgohar wrote: post #28

Thankfully all my nodes are back online and accessible via SSH but there are currently secondary effects due to the length of the outage that I am dealing with, such as my DNS server being deregistered with Namecheap from serving as a personal DNS server (their name for that feature). The consequence of this is also that my email is not currently work, since that affects the hostname the email server operates under s…

I'm in the exact same boat- mail server hosted on linode with namecheap as my registrar. I couldn't figure out why my DNS stopped working despite my linode finally coming back up, and... now I know where to look.

Re: Linode / Akamai US-EAST is down

#30

Post by mnordhoff »

basilgohar wrote: post #28

Thankfully all my nodes are back online and accessible via SSH but there are currently secondary effects due to the length of the outage that I am dealing with, such as my DNS server being deregistered with Namecheap from serving as a personal DNS server (their name for that feature). The consequence of this is also that my email is not currently work, since that affects the hostname the email server operates under s…

Wait, Namecheap deactivates your domain name if you have a temporary DNS outage?
Post reply on HN