Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

491–500 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#491
post #66

Earlier quoted context omitted.

If a fire supression kicks in or the fire department shows up with their hoses, would they still say the fire destroyed it or just say destroyed due to fire and water damage? Also, fire suppression system do fail. There was an infamous incident in LA for one of the studios. They built a warehouse to be a tape vault with tapes going back to the 80s. A fire started, but the suppression system failed because there was n…

It wasn’t just “tapes going back to the 80s.” Those were just the media Universal initially admitted to losing. No, that building was the mother lode. It had film archives going back over 50 years, and worst of all — unreleased audio tape masters for thousands upon thousands of recording artists. The amount of “remastered” album potential that fire destroyed is probably in the billions of dollars, let alone the histo…

This is why I'm such a fan of digitizing. If you have 1 film master you effectively have none. Do some 8K or 16K scans of that master and effectively manage the resulting data and you're effectively 100% immune from future loss in perpetuity.

Losing parts of history like that is tragic.

Re: Fire declared in OVH SBG2 datacentre building

#492
post #418

The key differentiator of OVH is the very compact datacenters they achieve thanks to water cooling. Some OVH exec were touting about that in a recent podcast. Interestingly in this case, having a very compact data center was probably an aggravating factor. This shows how complex these technical choices are, you have to think of operating savings, with a trade off on the gravity of black swan events...

Interesting. That said, the technique is not the issue here, losing a whole datacenter can always happen. This event would have been much less serious if all the four SBG* datacenters were not all so close to each other on the same plot of land. They are so close to each other that they are basically the same physical datacenter with 4 logical partitions.

They are all annexes built up over time and a victim of their own success (the site was meant to be more of a edge node in size). The container based annexes were meant to be dismantled 3 years ago but profit probably got in the way

Re: Fire declared in OVH SBG2 datacentre building

#493

Earlier quoted context omitted.

Yep you’re right, learning about this was part of some onboarding training we had to complete... it was an interesting proof of concept, but finding the right people to maintain the infra with both IT and Scuba skills was a narrow niche to nail down ;)

I don't opening it at any point before decomissioning it completely is even an afterthought with that. They just write off any failures and roll with it as long as it's viable.

Yeah that was one of the show-stopping issues, inability to repair / hotswap etc...

Re: Fire declared in OVH SBG2 datacentre building

#494

in case there were not posted before, here are pictures of SBG2 in flames taken by the firemen. https://twitter.com/xgarreau/status/1369559995491172354 This puts an image on the sentence "SBG2 is destroyed". Do not expect any recovery from SBG2.

Status shows greens across the board

http://status.ovh.com/vms/index_sbg2.html

Am I looking at the wrong thing or am I right to wonder why we still bother with public status pages if it never shows the real status?

Edit: nvm just saw another comment pointing out the same further down the thread (I randomly came across this page while looking for the physical location of another DC)

Re: Fire declared in OVH SBG2 datacentre building

#496

Earlier quoted context omitted.

IMHO, the part they had no plan for was being unable to just require their employees to come in anyway...

The more insecure your workers, the easier it is to get them to come in, regardless of what the supposed rules may or may not be. Fast Fashion for example often employs workers in more or less sweatshop conditions close to the customers (this makes commercial sense, if you make the hot new items in Bangladesh you either need to expensively air freight them to customers or they're going to take weeks to arrive after t…

Just to clarify: trans-atlantic shipments take a week port-to-port, e.g. Newark, NJ, USA to Antwerp, Belgium. (Bangladesh to Italy via Suez-channel looks like a 2-week voyage, or 3 weeks to the US west coast. Especially the latter would probably have quite a few stops on the way along the Asian coast.) You get better economics than shipping via air-freight from one full pallet and up. Overland truck transport to and from the port is still cheaper than air freight, at least in the US and central Europe.

For these major routes, there are typically at least bi-weekly voyages scheduled, so for this kind of distance, you can expect about 11 days pretty uniformly distributed +-2 days, if you pay to get on the next ship.

This may lead to (committing to) paying for the spot on the ship when your pallet is ready for pickup at the factory, not when it arrives at the port) and use low-delay overland trucking services. Which operate e.g. in lockstep with the port processing to get your pallet on the move within half a day of the container being unloaded from the ship, ideally having containers pre-sorted at the origin to match truck routes at the destination. So they can go on a trailer directly from the ship and rotate drivers on the delivery tour, spending only a few minutes at each drop-off.

Because those can't rely on customers to be there and get you unloaded in less than 5 minutes, they need locations they can unload at with on-board equipment. They'd notify the customer with a GPS-based ETA display, so the customer can be ready and immediately move the delivery inside. Rely on 360-degree "dashcam" coverage and encourage the customer to have the drop-off point under video surveillance, just to easily handle potential disputes. Have the delivery person use some suitable high-res camera with a built-in light to get some full-surface-coverage photographic evidence of the condition it was delivered in.

I'd guess with a hydraulic lift on the trailer's back and some kind of folding manual pallet jack stuck on that (fold-up) lift, so they drive up to the location, unlock the pallet jack, un-fold the lift, lower the lift almost to the ground, detach the pallet jack to drop it the last inch/few cm to the ground, pull the jack out, lower the lift the rest of the way, drive it on to the lift, open the container, get up with the pallet jack, drive the pallets (one-by-one) for this drop-off out of the container and leave them on the ground, close and lock the container, re-arm the jack's hooks, shove it jack back under the slightly-lowered folding lift, make it hook back in, fold it up, lock the hooking mechanism (against theft at a rest stop (short meal and toilet breaks exist, but showering can be delayed for the up to 2 nights)), fold it all the way up, and go on to drive to their next drop-off point.

Re: Fire declared in OVH SBG2 datacentre building

#497

Back in the late 90s, I implemented the first systematic monitoring of WalMart Store's global network, including all of the store routers, hubs (not switches yet!) and 900mhz access points. Did you know that WalMart had some stores in Indonesia? They did until 1998. So when https://en.wikipedia.org/wiki/May_1998_riots_of_Indonesia started happening, we heard some harrowing stories of US employees being abducted, amon…

Thanks for sharing this interesting story. Part of my family immigrated from Indonesia due to those riots, but I was unaware up until today of the details covered by the Wikipedia article you linked. I remember during the 2000s and 2010s that WalMart in the USA earned a reputation for it's inventories primarily consisting of Chinese-made goods. I'm not sure if that reputation goes all the way back to 1998, but it mak…

I can't recall (and probably didn't know at the time...it was far from my area) where products were sourced for the Indonesia stores.

Prior to the early 2000s, WalMart had a strong 'buy American' push. It was even in their advertising at the time, and literally written on the walls at the home office in Bentonville.

Realities changed, though, as whole classes of products were more frequently simply not available from the United States, and that policy and advertising approach were quietly dropped.

Just for the hell of it, I did a quick youtube search: "walmart buy american advertisement" and this came up: https://www.youtube.com/watch?v=XG-GqDeLfI4 "Buy American - Walmart Ad". Description says it's from the 1980s, and that looks about right.

Re: Fire declared in OVH SBG2 datacentre building

#498
post #180

Earlier quoted context omitted.

They did not have a fire suppression system, only smoke detection. So yeah, they missed something.

There are photos of the fire suppression system in one of their data centres in this (French) forum thread: https://lafibre.info/ovh-datacenter/ovh-et-la-protection-inc... . They have sprinklers, with the reasoning being that the burning racks' data is gone anyway if there's a fire, and at least sprinklers don't accidentally kill the technicians.

> They have sprinklers, with the reasoning being that the burning racks' data is gone anyway if there's a fire

I think the real problem, per that post, is this:

>> They are simple sprinklers that spray with water. It has nothing to do with very high pressure misting systems, where water evaporates instantly, and which save servers. Here, it's watering, and all the servers are dead. It's designed like that. Astonishing, isn't it?

>> Obviously, they rely above all on humans to extinguish a possible fire, unlike all conventional data centers.

(all thanks to Google Translate)

This strikes me as a terrible safety system because even if a human managed to detect the fire, they have to make a big call: is the risk of flooding the facility and destroying a ton of gear worth putting out a fire? By the time the human decides "yes, it is", it may well be too late for the sprinklers.

> and at least sprinklers don't accidentally kill the technicians.

Not a real risk with modern 1:1 argon:nitrogen systems – the goal is to pump in inert gases and reduce oxygen content to around 13%, a point where the fire is suppressed and people can survive. You wouldn't want to be in a room breathing 13% oxygen for a long time, but it won't kill you.

All in all, it looks like this was a "normal accident"[1] for a hosting company that aggressively competes on price. The data center was therefore built with less expensive safeties and carried a higher risk of catastrophic failure.

[1]: https://en.wikipedia.org/wiki/Normal_Accidents

Re: Fire declared in OVH SBG2 datacentre building

#499
post #180
post #175

Earlier quoted context omitted.

OVH design their own datacenters, so it's possible that they missed something or some system or another didn't work as intended, thus the heavy damage.

They did not have a fire suppression system, only smoke detection. So yeah, they missed something.

> They did not have a fire suppression system

I find it very hard to believe that that would pass code anywhere in the US/EU or most of the world. They may not have had sprinklers but that doesn't mean there isn't fire suppression.

Post reply on HN