Late to the party, but ... context? Everyone is talking like OVH or the SBG2 buildings are well-known and common knowledge.
Fire declared in OVH SBG2 datacentre building
561–570 of 613 posts
Re: Fire declared in OVH SBG2 datacentre building
#562Interesting. I wonder if the cladding was a major problem here? It looks like it has all burnt out and could have had the fire spread extremely rapidly on the outside.
The cladding was metal so very unlikely it contributed to the fire spreading.
Re: Fire declared in OVH SBG2 datacentre building
#563Earlier quoted context omitted.
Much cheaper and better performance at the high end. Doesn't compete at all at the low-end, except through their budget brand Kimsufi. I don't see them really as targeting the same market.
I rent a server from OVH for $32 a month. It's their So You Start line... doesn't come with fancy enterprise support and the like. It's a 4 core 8 thread Xeon with 3x 1TB SATA with 32GB of ECC RAM IIRC (E3-SAT-1-32, got it during a sale with a price that is guaranteed as long as I keep renewing it) The thing is great, I can run a bunch of VM's on it, it runs my websites and email. Overall to get something comparable…
Yeah, I forgot they also have the so you start brand. It's probably more expensive than the majority of what digital ocean sells, but there is some overlap for sure.
Re: Fire declared in OVH SBG2 datacentre building
#564Earlier quoted context omitted.
They're separate buildings, with separate power systems, just standing next to another. Next to SBG2 are also SBG3 and SBG4.
Those buildings weren't quite as far apart as they should have been if a fire in one requires all 4 of the others to be turned off...
Apparently hey are more like (temporary) extensions of the main building than separate DCs.
Re: Fire declared in OVH SBG2 datacentre building
#565Earlier quoted context omitted.
They're separate buildings, with separate power systems, just standing next to another. Next to SBG2 are also SBG3 and SBG4.
Those buildings weren't quite as far apart as they should have been if a fire in one requires all 4 of the others to be turned off...
Apparently they are more like (temporary) extensions of the main building than separate DCs.
Re: Fire declared in OVH SBG2 datacentre building
#566Earlier quoted context omitted.
If you need that large machines, and are willing to use reserved instances, you'd go with dedis on OVH instead of VMs. Which is significantly cheaper
OVH dedicated instances start at about the size of an a1.metal instance, which is ~30% more than the comparable OVH instance, but you can get discounts in various ways. Or you could use t4g.2xlarge, which is cheaper. There's no situation where OVH is 3x cheaper (I mean maybe if bandwidth is your thing, but IDK).
Performance of those AWS instances comes nowhere close despite the specs.
Re: Fire declared in OVH SBG2 datacentre building
#567Unfortunately A lot of people are going to find out the hard way today why AWS/GCP/Big Expensive Cloud is so expensive (Hint: they have redundancy and failover procedures which drive up costs). Keep in mind I’m talking not of “downtime” but of actual data loss which might affect business continuity. This is really tragic. I’m hoping they have some kind of multi regional backup/replication and not just multi zones (al…
Yup, I've never heard of a fire taking out a Big Cloud DC. They actually know what they're doing and don't put server racks in shipping containers stacked on top of each other. If you want quality in life, sometimes you have to pay for it. Personally I'll continue to use these third world cloud providers. But I like to live on the edge.
Re: Fire declared in OVH SBG2 datacentre building
#568The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…
> What's YOUR Disaster Recovery Plan?
Prayer and hope, usually.
Re: Fire declared in OVH SBG2 datacentre building
#569Earlier quoted context omitted.
Well, yeah, these normal inert gas fire suppression systems don't do a good job if humans can still breathe. The Novec 1230 based ones can actually be sufficiently effective for typical flammability properties you can cheaply adhere to in a datacenter, but even then you iirc would want to add both that and some extra oxygen, because the nitrogen in the air is much more effective at suffocating humans than at suffocat…
> I mean, in principle, if you gave re-breathers to the workers and have some airlocks, you could afford to keep that atmosphere permanently, At this point I feel like it would be cheaper just to not have workers go there. Fill the place completely full of nitrogen with an onsite nitrogen generator (and only 1atm pressure). Have 100% of regular maintenance and as much irregular maintenance as possible be done by robo…
That seems reasonable.
But just to clarify what I meant with airlock: some thick plastic bag with a floor-to-ceiling zipper on the "inner" and "outer" ends, and for entry, it's first collapsed by a pump sucking the air out of it. Then you open the zipper on the outer end, step in, close the zipper, let the pump suck away the air around you, and open the inner zipper (they should probably be automatically operated, as you can't move well/much when you are "vacuum bagged").
For exit, basically just the reverse, with the pump pumping the air around the person to wherever the person came from.
The general issue with unbreathable atmospheres is that a failure in their SCBA gear easily kills them.
And re-breathers that are only there so you don't have to scrub the atmosphere in the room as often shouldn't be particularly expensive. You may even get away with just putting a CO2 scrubber on the exhaust path, and giving them slightly oxygen-enriched bottled air so you can keep e.g. a 50:50 oxygen:nitrogen ratio inside (so e.g. 20% O2, 20% N2, 60% Novec 1230). And it doesn't even need to be particularly effective, as you can breathe in quite a bit of the ambient air without being harmed, and the environment can tolerate some of your CO2. Like, as long as it scrubs half of your exhausted CO2 it won't even feel stuffy in there (you could handle the ambient air you'd have without re-breathers being used, as it'd be just 1.6% CO2, but you'd almost immediately get a headache).
They'd have an exhaust vent for pressure equalization, which would chill the air to condense and re-cycle the Novec 1230. For pressure equalization in the other direction, they'd probably just boil off some of that recycled Novec 1230.
So yeah, re-breather not needed, if you just get a mouth+nose mask to breathe bottled 50:50 oxygen:nitrogen mix. That 50% oxygen limit (actually 500 mBar) is due to long-term toxicity, btw. Prolonged exposure to higher levels causes lung scarring and myopia/retina detachment, so not really fun.
Re: Fire declared in OVH SBG2 datacentre building
#570Earlier quoted context omitted.
What the hell, here's another story. The summary to catch your attention: in the early 2000s, I first became aware of WalMart's full scale switch to product sourcing from China by noting some very unusual automated network to site mappings. Part of what my team (Network Management) did was write code and tools to automate all of the various things that needed to be done with networking gear. A big piece of that was a…
In the early 2000s I was working as a field engineer installing/replacing/fixing network equipment for Walmart at all hours. It's pretty neat to hear the other side of the process! If I remember correctly there was some policy that would automatically turn off switch ports that found new, unrecognized devices active on the network for an extended period of time, which meant store managers complaining to me about voip…
Ah neat, so you were an NCR tech! (I peeked at your comment history a bit.) My team and broader department spent a lot of hours working with, sometimes not in the most friendly terms, people at different levels in the NCR organization.
You're correct, if Drake (the always running discovery engine) didn't detect a device on a given port over a long enough time, then another program would shut that port down. This was nominally done for PCI compliance, but of course having open, un-used ports especially in the field is just a terrible security hole in general.
In order to support legit equipment moves, we created a number of tools that the NOC and I believe Field Support could use to re-open ports as needed. I think we eventually made something that authorized in-store people could use too.
As an aside, a port being operationally 'up' wasn't by itself sufficient for us mark the port as being legitimately used. We had to see traffic coming from it as well.
You mentioned elsewhere that you're working with a big, legacy Perl application, porting it to Python. 99% of the software my team at WalMart built was in Perl. (: I'd be curious to know, if you can share, what company/product you were working on.