the last outage was in Fremont as well. I think I'll be moving my servers to Dallas or another provider..
Linode Fremont servers are down
11–20 of 32 posts
Re: Linode Fremont servers are down
#12Networking outage again. Sigh. So glad I just moved my prod servers to gcloud last week... To be fair to Linode, the Fremont outages always seem to be caused by HE.net (the "upstream provider"). Used to co-lo in the same datacenter and had issues like this constantly. Not sure what HE's problem is.
The right solution is to get two upstreams and BGP. ... which is unfortunately rather more difficult when you are in a datacenter owned by he.net :( It's possible, though, and I thought linode had done so, I think XO communications is in that building (that could be the other fremont he.net data center, it's been a while. Who owns XO these days? Verizon. I have been out for a while.) I was in that same datacenter for…
Identified - At this time we have identified the connectivity issues affecting our Fremont data center as
being the result of a power outage. Our team is working as quickly as possible to have connectivity restored.
We will provide additional updates as they develop.
Well...Re: Linode Fremont servers are down
#13Looks like it's just come back, one of my servers just woke up.
Re: Linode Fremont servers are down
#14This is kinda what a status page is for, not the HN front page - linked being down is nothing new - it’s why we moved our stack away to AWS, what, seven years ago? Well, that and the horribly overloaded switches and poor implementation of the Xen packet scheduler that caused unacceptable latency spikes to memcached. It’s great for hosting toy sites that sit on a single box and don’t have an uptime SLA - and people sa…
Re: Linode Fremont servers are down
#15Re: Linode Fremont servers are down
#16Re: Linode Fremont servers are down
#17Networking outage again. Sigh. So glad I just moved my prod servers to gcloud last week... To be fair to Linode, the Fremont outages always seem to be caused by HE.net (the "upstream provider"). Used to co-lo in the same datacenter and had issues like this constantly. Not sure what HE's problem is.
The right solution is to get two upstreams and BGP. ... which is unfortunately rather more difficult when you are in a datacenter owned by he.net :( It's possible, though, and I thought linode had done so, I think XO communications is in that building (that could be the other fremont he.net data center, it's been a while. Who owns XO these days? Verizon. I have been out for a while.) I was in that same datacenter for…
Re: Linode Fremont servers are down
#18Earlier quoted context omitted.
The right solution is to get two upstreams and BGP. ... which is unfortunately rather more difficult when you are in a datacenter owned by he.net :( It's possible, though, and I thought linode had done so, I think XO communications is in that building (that could be the other fremont he.net data center, it's been a while. Who owns XO these days? Verizon. I have been out for a while.) I was in that same datacenter for…
Yup, and looks like this was caused by another power outage... had the same thing happen when I was there. Surprised they still haven't added proper backups after all these years.
Personally, if I were building a cut-rate datacenter with used/unreliable power equipment? I'd build two parallel power systems to each rack. Each customer would get two independent power feeds (and stern warnings to not use more than 80% of one circuit total) (I mean, while new kit is super expensive, used power conditioning equipment can be had on the cheap if you are in the right place at the right time.)
Then, so long as customer buys dual PSU equpment (or their own small ATS, and I've had better luck with 20a automatic transfers switches) they'd be fine.
Of course, I've never gotten the chance to try, (well, I did kind of have a chance once, but I kind of blew it for non-technical reasons before I got to build out a power system of my liking.) so maybe I'm missing something. Maybe the electrician work is more expensive than I think or something.
I'm a little surprised that anyone with any scale still uses he.net for physical hosting; Last time I tried to negotiate with them, at my scale (20kw) they essentially charged $400 for every 1.4kw, (and they gave you a rack with that 1.4kw) which is kindof a terrible deal compared to higher density co-location providers like coresite or BAIS, where your racks are closer to 5kw each, but that rack usually comes closer to $1300 or so.
(Most rackmount servers are power-dense enough to eat 5Kw of power in 42u, with some space over)
This means that after your first few servers, the higher end datacenters are actually way cheaper.
I mean, don't get me wrong, I think he.net bandwidth is a great part of anyone's BGP mix, and I think that he.net hosting is the best deal around if you only have 1.4kw of power and want your own rack (often they will sell you your own rack, 1.44kw of power, and a nice unmetered gigE pipe for $500 a month, which is a killer deal for a rack and a gig. But it's kinda terrible if you are using serious amounts of power)
On the other hand, the he.net remote hands are... some of the best. I mean, they are remote hands and super junior people, with all the problems that come with that, (remote hands is, as far as I can tell, a junior sort of job many places I've seen.) but they are on site, on the ball, and every time I called in, they were right there, and it's free.
Last time I called coresite for remote hands, they said they would charge me $200/hr... reasonable, I suppose, for getting someone junior-ish out of bed, but they had to page the person, and their person still wasn't there two hours later when I was able to arrive.
Re: Linode Fremont servers are down
#19Networking outage again. Sigh. So glad I just moved my prod servers to gcloud last week... To be fair to Linode, the Fremont outages always seem to be caused by HE.net (the "upstream provider"). Used to co-lo in the same datacenter and had issues like this constantly. Not sure what HE's problem is.