Does it really take over an hour just to figure out why a whole data centre went dark?
Well there you go it's a power outage.
Linode Fremont servers are down
21–30 of 32 posts
Re: Linode Fremont servers are down
#22the last outage was in Fremont as well. I think I'll be moving my servers to Dallas or another provider..
Yeah, I am getting hit by this too. I still want to support Linode but can't if this is going to be a regular occurrence. How long ago was the last outage?
about 3 months ago: https://news.ycombinator.com/item?id=16448100
Re: Linode Fremont servers are down
#23It seems this was just FMT1 that went dark
Re: Linode Fremont servers are down
#24Reddit also had trouble about an hour ago, for about 15 minutes I think. Wonder if it's related.
Re: Linode Fremont servers are down
#25Earlier quoted context omitted.
Yup, and looks like this was caused by another power outage... had the same thing happen when I was there. Surprised they still haven't added proper backups after all these years.
when I was at he.net, the outage reports were all about ATS failures and the like. I mean, switching that much power is serious busines... a lot of the cut-rate datacenters I've been at mostly had power outages when their switching equipment failed, even when they had decent UPS and generator capacity. Personally, if I were building a cut-rate datacenter with used/unreliable power equipment? I'd build two parallel po…
most (normal?) datacenters have A/B/C power feeds. Also east/west network runs. even if you don't have multi-line PSU's you can at least stagger the racks. part of being cut-rate is that they don't have these things :(
Re: Linode Fremont servers are down
#26This is kinda what a status page is for, not the HN front page - linked being down is nothing new - it’s why we moved our stack away to AWS, what, seven years ago? Well, that and the horribly overloaded switches and poor implementation of the Xen packet scheduler that caused unacceptable latency spikes to memcached. It’s great for hosting toy sites that sit on a single box and don’t have an uptime SLA - and people sa…
Well yeah but then where do you tell your story about moving to AWS?
If it gets upvoted enough to hit the front page enough people must be interested in it - there's a hard low-end cap if I recall, must have at least x votes before it's even considered for front page
For popular services such as Linode it also makes sense as if you're ever thinking "Man I see this all the time on HN" it's probably a good call to avoid that provider
Re: Linode Fremont servers are down
#27Earlier quoted context omitted.
when I was at he.net, the outage reports were all about ATS failures and the like. I mean, switching that much power is serious busines... a lot of the cut-rate datacenters I've been at mostly had power outages when their switching equipment failed, even when they had decent UPS and generator capacity. Personally, if I were building a cut-rate datacenter with used/unreliable power equipment? I'd build two parallel po…
most (normal?) datacenters have A/B/C power feeds. Also east/west network runs. even if you don't have multi-line PSU's you can at least stagger the racks. part of being cut-rate is that they don't have these things :(
corsite (and all but the lowest end datacenters) will sell you multiple power feeds... but you usually have to pay substantially more for those feeds every month than you would pay for the same watts out of a single feed, and my experience at the newer coresite locations is that the single feed is reliable enough. (when I was in charge of prgmr, we moved from he.net with single feeds... well, we ended up at coresite santa clara with single feeds. I think there hasn't been a datacenter fault power outage at coresite, in like 7 years, even though I was on a single feed. There were a few outages that were my fault, but that's a different thing.)
I spent some time in the (I think now defunct) SVTIX co-lo at 250 stockton. It was super low-end, super shoestring, in some ways lower-end than he.net - but they had multiple power systems and would sell you a/b power if you wanted to pay enough for it. 250 stockton was certainly built out to higher standards than the he.net datacenters, though the he.net datacenters were better maintained.
(My favorite "my fault" power outage story was that I would give my friends co-lo access, right? And then I'd call them when I needed remote hands that knew what they were doing. Well, you know how cheap nerds are. One of my buddies shows up with one of those ancient PSUs that have a 110-220 votage switch. My rack was 208v. He had the switch on the 110 setting. It fried my PDU, the same avocent that everyone in the rack was on. The outage wasn't long; I had a spare in the office that was only a few minutes away. It was super annoying, 'cause I had earlier given the man a perfectly good 1u chassis that came with a modern 100-240v autoswitch psu... which he put in. This was his second computer; I would have been happy to give him another chassis/PSU. I was so mad. So mad!
I actually didn't see him for a while... the next time I saw him... I was at a party, telling this very story. He walks in right in the middle. I point and kinda shout 'It was him!" which, yeah. uh, I should not have done that.)
Re: Linode Fremont servers are down
#28This is kinda what a status page is for, not the HN front page - linked being down is nothing new - it’s why we moved our stack away to AWS, what, seven years ago? Well, that and the horribly overloaded switches and poor implementation of the Xen packet scheduler that caused unacceptable latency spikes to memcached. It’s great for hosting toy sites that sit on a single box and don’t have an uptime SLA - and people sa…
You do realise that this post is a link to the Linode status page. I'm not sure what you're saying in that first sentence.
All I ever hear about Linode is their great speeds but also their downtime issues. I'm still on DigitalOcean and Vultr. AWS is too expensive for what I use it for right now.
Re: Linode Fremont servers are down
#29The funny thing is that our ex CTO thinks that using AWS has more disadvantages than having all of our servers in two different regions in Linode. Mind you, he also thought that CDNs were not as useful as serving assets statically from those two servers as well.
If you use two different Linode regions this outage shouldn’t affect you at all?
The CDN thing is a pet peeve of mine too. Are you even measuring load performance? Where are your customers located? If you’re just wanting to throw everything on a CDN “because it’s faster” then no it’s not very useful at all.
Re: Linode Fremont servers are down
#30Networking outage again. Sigh. So glad I just moved my prod servers to gcloud last week... To be fair to Linode, the Fremont outages always seem to be caused by HE.net (the "upstream provider"). Used to co-lo in the same datacenter and had issues like this constantly. Not sure what HE's problem is.
And to be fair to HE, I've been coloing with them for years now (in FMT2) and I'm yet to have any problems.