Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

Post reply on HN
Print view View original

101–110 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#101

Post by Twirrim »

sparrish wrote: post #38
geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

Got burned once (no pun intended), learned my lesson. Hot spare on a different continent with replicated data along with a third box just for backups. The backup box gets offsite backups held in a safe with another redundant copy in another site in another safe. Restores are tested quarterly. Keep backups of backups. Once bitten, twice shy.

> Hot spare on a different continent

Just be cautious about data locality laws (not likely to affect you as joe average, more for businesses)

Re: Fire declared in OVH SBG2 datacentre building

#102

Post by qwertykb »

bithaze wrote: post #94
qwertykb wrote: post #91

Honestly just for the memes. https://isovhonfire.com

Probably not a good look for this to be the first thing another hosting company thinks to put up in response.

/shrug/ We have our infrastructure partially in OVH, I see it as a friendly jab at them and a way to get updates without having to navigate to twitter.

Re: Fire declared in OVH SBG2 datacentre building

#104

Post by raverbashing »

paulmd wrote: post #89
kijin wrote: post #14

Earlier quoted context omitted.

One of my backup servers used to be in the same datacenter as the primary server. I only recently moved it to a different host. It's still in the same city, though, so I'm considering other options. I'm not a big fan of just-make-a-tarball-of-everything-and-upload-it-to-the-cloud backup methodology, I prefer something a bit more incremental. But with Backblaze B2 being so cheap, I might as well just upload tarballs t…

> I'm not a big fan of just-make-a-tarball-of-everything-and-upload-it-to-the-cloud backup methodology, I prefer something a bit more incremental. pretty much a textbook use-case for zfs with some kind of snapshot-rolling utility. Snap every hour, send backups once a day, prune your backups according to some timetable. Transfer as incrementals against the previous stored snapshot. Plus you get great data integrity ch…

You can do the same with Ext4

Re: Fire declared in OVH SBG2 datacentre building

#105

Post by raverbashing »

rgj wrote: post #103

Video here. This will be Disaster Recovery 101 material. https://mobile.twitter.com/abonin_DNA/status/136953802824345...

Ouch, that's not pretty... but it seems that the fire was constrained to 1 or 2 sectors (inside the building) - per their updates

Not sure how good were the fire suppression systems of the building.

Re: Fire declared in OVH SBG2 datacentre building

#106

Post by rgj »

plasma wrote: post #97

“ Update 7:20am Fire is over. Firefighters continue to cool the buildings with the water. We don’t have the access to the site. That is why SBG1, SBG3, SBG4 won’t be restarted today.” https://mobile.twitter.com/olesovhcom/status/136953578757072...

With water, they said.

Re: Fire declared in OVH SBG2 datacentre building

#107

Post by el-salvador »

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

Pictures from the fire:

https://www.dna.fr/amp/faits-divers-justice/2021/03/10/stras...

Re: Fire declared in OVH SBG2 datacentre building

#108

Post by slaymaker1907 »

dylan604 wrote: post #56
sverhagen wrote: post #41

Earlier quoted context omitted.

> probably That makes it sound like you didn't try/practice. I imagine that in a real-life scenario things will be a little more painful than in one's imagination.

Exactly. Having a plan is only part of it. Good disaster plans do dry runs a couple of times a year (when time changes is always convenient reminder). If you rehearse the recovery when you're not panicked, you have a better chance of not skipping a step when the timing is much more crucial. Also, some sort of guide with steps given procedurally is a great idea.

I don't think this is necessarily true for all parts of a disaster plan. Some mechanisms may be untestable because it is unknown how to actually trigger it (think certain runtime assertions, but on a larger scale).

Even if it possible to trigger and test, actually using the recovery mechanism may have some high cost either monetarily or maybe losing some small amount of data. These mechanisms should almost always be an additional layer of defense and only be invoked in case of true catastrophe.

In both cases, the mechanisms should be tested as thoroughly as possibly, either through artificial environments that can simulate improbable scenarios or in the latter case on a small test environment to minimize cost.

Re: Fire declared in OVH SBG2 datacentre building

#109

Post by jrockway »

mhh__ wrote: post #55
lgleason wrote: post #7

https://www.youtube.com/watch?v=1EBfxjSFAxQ

I'm not sure if it's because my tolerance of Graham Linehan has snapped or not, but I barely laugh at the IT Crowd any more. As with other GL shows I find it's just mostly held together but the cast's delivery and such The laugh track and the writing is honestly dated even by the standards of Dads Army.

I don't remember the details, but I think that season 2 kind of retroactively ruined season 1. They used to have all those O'Reilly and EFF stickers, and working at a help desk at the time, it felt very authentic. Then everything got super nice in season 2 -- leather couches, people were dressing nicely, etc. It kind of lost its charm. You can't rewatch it because you know Denholm is just going to randomly jump out of a window.

(Having said that, I think "Fire" was a memorable episode that is still amusing. The 0118911881999119 song, "it's off, so I'll turn it on... AND JUST WALK AWAY".)

It might have been ahead of its time. Silicon Valley was well received and is as nerdy and intricately detailed as Season 1 of the IT Crowd. "Normal people" thought it was far out and zany. People that work in tech have been to all those meetings. And, a major character was named PG!

Re: Fire declared in OVH SBG2 datacentre building

#110

Post by jhugo »

NorwegianDude wrote: post #90

I can't see anything about a fire suppression system mentioned? Doesn't OVH have one, except for colocation datacenters? A fire detection system using eg. lasers and Inergen(or Argonite) for putting the fire out is commonly used in datacenters. The gas fills the room and reduces the amount of oxygen in the room so most fires are put out within a minute. The cool thing is that the gas is designed to be used in rooms w…

At a datacentre I used to visit years ago, part of the site induction was learning that if the fire suppression alarm went off, you had a certain amount of time to get out of the room before the argon would deploy, so you should always have a path to the nearest exit in mind. The implication was that it wasn't safe to be in the room once it deployed, but I don't know for sure.
Post reply on HN