Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

Post reply on HN
Print view View original

571–580 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#571

Post by Scoundreller »

PudgePacket wrote: post #330
Ploskin wrote: post #313

Earlier quoted context omitted.

Is the status page relevant though? At the very least, OVH immediately made a status announcement on their support page and they've been active on Twitter. I don't see anything shady here. From their support page: > The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4. If your production is in Strasbourg, we recommend to activate your Disaster Recovery Plan What more could you…

> Is the status page relevant though? What's the point of a status page then if it does not show you the status? I don't want to be chasing down twitter handles and support pages during an outage.

Still better than Amazon where their status page describes little and fat chance of anyone official sharing anything on social media either.

I wonder if a server fire would cause Amazon to go to status red. So far anything and everything has fallen under yellow.

Re: Fire declared in OVH SBG2 datacentre building

#572

Post by Scoundreller »

dylan604 wrote: post #453
lrem wrote: post #164

Earlier quoted context omitted.

There's a problem with testing sprinklers: engaging them can be damaging to contents and even structures. So, we're talking about completely emptying the facility, then taking it offline to dry for a time. I've never heard about this being done to anything that was already operational (but I wasn't researching this either).

There are methodds of testing the water pressure in the pipes without actually engaging the sprinkler heads. It is part of the normal checks done during the maintenence/inspection a business is supposed to have done. In fact, one place I was in had sensors, and would sound the actual fire alarm if the pressure fell below tolerance at any time. The lack of pressure in the Universal vault was 100% unexcusable.

It’s common for sprinklers in parking garages in cold climates to be “dry” where there’s a bubble of air that needs to run through first before water shoots out from a non-freezeable source.

Want to test the system? Just turn off that water valve, hook up your air compressor and pressure test to your heart’s content.

Re: Fire declared in OVH SBG2 datacentre building

#573

Post by Scoundreller »

thspimpolds wrote: post #400
jhugo wrote: post #220

Earlier quoted context omitted.

Yeah, that was always my understanding. GP saying "the gas is designed to be used in rooms with people, so that [it] can be triggered any time" made me second-guess that though. Maybe there is a concentration of oxygen that is high enough for humans to survive, yet too low to sustain combustion?

The "gas" is more like a fog of capsules. FM-200 is a common one. Basically it has a Fire suppression agent inside crystals which are blasted into the room by compressed air. These crystals melt when they get over a certain temperature and therefore won't kill you; however, breathing that in isn't really pleasant. Source: I've been in an FM-200 discharge

Does everyone carry an N95 or equivalent? I guess many of us have gotten good at donning them by now...

Re: Fire declared in OVH SBG2 datacentre building

#574

Post by Scoundreller »

monsieurbanana wrote: post #134
NorwegianDude wrote: post #90

I can't see anything about a fire suppression system mentioned? Doesn't OVH have one, except for colocation datacenters? A fire detection system using eg. lasers and Inergen(or Argonite) for putting the fire out is commonly used in datacenters. The gas fills the room and reduces the amount of oxygen in the room so most fires are put out within a minute. The cool thing is that the gas is designed to be used in rooms w…

> even destroying harddrives So loud that it destroys hard drives... That's scary, are people's eardrums much more resistant?

A lot of DCs are loud enough that it’s quite pleasant to wear ear protection. Even better with headphones that play what you want to listen to.

Re: Fire declared in OVH SBG2 datacentre building

#575

Post by gadrev »

nikanj wrote: post #146

According to the official status page, the whole datacenter is still green http://status.ovh.com/vms/index_sbg2.html

Noticed the same this morning, but note they took the link out of the main page (of SBG2 and 3): https://www.ovh.com/world/community/status/

Probably the best they could do at the moment.

Re: Fire declared in OVH SBG2 datacentre building

#576

Post by Crosseye_Jack »

avereveard wrote: post #549
Diederich wrote: post #443

Back in the late 90s, I implemented the first systematic monitoring of WalMart Store's global network, including all of the store routers, hubs (not switches yet!) and 900mhz access points. Did you know that WalMart had some stores in Indonesia? They did until 1998. So when https://en.wikipedia.org/wiki/May_1998_riots_of_Indonesia started happening, we heard some harrowing stories of US employees being abducted, amon…

ovh monitoring is even better, everything is green on the destroyed dc http://status.ovh.com/vms/index_sbg2.html

That’s because it runs on the cloud… :-p

Re: Fire declared in OVH SBG2 datacentre building

#577

Post by jaza »

helge9210 wrote: post #544

Almost 11 years ago (March 27 2010) in Ukraine datacenter of company Hosting.ua went in flames as clients were watching their systems go unresponsive at various rows of the datacenter. Anti-fire systems didn't kick-in. The reason? For couple of days the system was detecting a little bit of smoke from one of the devices. Operators weren't able to pinpoint exact location, considered it a false alarm and manually switch…

Sounds disturbingly similar to the story of how Chernobyl went into meltdown. Also in Ukraine!

Re: Fire declared in OVH SBG2 datacentre building

#578

Post by kakwa_ »

ev1 wrote: post #447
treesknees wrote: post #446

Earlier quoted context omitted.

Is that a typo? I only see OVH bare metal starting at >$50. How could a provider offer a bare metal server for $5?

OVH has new servers. Their sub-brand soyoustart has older servers (that are still perfectly fine), roughly E3 Xeon/16-32GB/3x2TB to 4x2TB for $40/m ex vat. Their other sub brand kimsufi for personal servers has Atom low-power bare metal with 2TB HDD (in reality it is advertised 500GB/1TB, but they don't really have any of those in stock left, if your drive fails they replace it with a 2T - so far this has been my exp…

Just as a side note, the name "kimsufi" comes from the French "qui me suffit", which roughly translate to "enough for me".

Re: Fire declared in OVH SBG2 datacentre building

#579

Post by hobojones »

Diederich wrote: post #570
hobojones wrote: post #554

Earlier quoted context omitted.

In the early 2000s I was working as a field engineer installing/replacing/fixing network equipment for Walmart at all hours. It's pretty neat to hear the other side of the process! If I remember correctly there was some policy that would automatically turn off switch ports that found new, unrecognized devices active on the network for an extended period of time, which meant store managers complaining to me about voip…

Ah neat, so you were an NCR tech! (I peeked at your comment history a bit.) My team and broader department spent a lot of hours working with, sometimes not in the most friendly terms, people at different levels in the NCR organization. You're correct, if Drake (the always running discovery engine) didn't detect a device on a given port over a long enough time, then another program would shut that port down. This was…

The NCR/Walmart relationship was fairly strained during my tenure. Given the sheer number of stores/sites that Walmart had and NCR's own problems, it was not always possible to provide the quality of service people might expect, especially with a smile. From a FE perspective, working on networking gear at Walmart meant that you were out at 11pm at night (typically after working 10-11 hours already and spending an hour or two onsite waiting for the part to arrive via courier) and your primary concern was to get the job done and get back home. The worst was to plug a switch in, watch it not power up, and realize you'd need to be back in the same spot at three or four hours later to try again.

Walmart must have been an interesting place to work during the late 90s, early 2000s - I imagine that most everywhere they had to solve problems at scale before scale was a considered a thing. I'd be very interested to see how the solutions created in that period match to best-practices today, especially since outside of the telecom or perhaps defense worlds there probably wasn't much prior art.

As for the Perl application, I probably shouldn't say since I'm still employed at the same company and I know coworkers who read HN. If you're interested, DM me and I can at least provide the company name and some basic details.

Re: Fire declared in OVH SBG2 datacentre building

#580

Post by ta988 »

extrasolar wrote: post #526
ta988 wrote: post #410

Earlier quoted context omitted.

Same but with a regular offline physical copy (cheap nas). One of my worries is a malicious destruction of the backups if anything worms its way in my network

hetzner has storage boxes and auto snapshots. so even if someone deletes the backups remotely there are still snapshots which they can't get unless they have control panel access.

My threat model is someone that would have full access to my computer without me knowing. So they could over time get access to passwords, modify my OS to MITM yubikeys... Over cautious really likely, but that doesnt cost me much more
Post reply on HN