Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

Post reply on HN
Print view View original

191–200 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#191

Post by geococcyxc »

raverbashing wrote: post #104
paulmd wrote: post #89

Earlier quoted context omitted.

> I'm not a big fan of just-make-a-tarball-of-everything-and-upload-it-to-the-cloud backup methodology, I prefer something a bit more incremental. pretty much a textbook use-case for zfs with some kind of snapshot-rolling utility. Snap every hour, send backups once a day, prune your backups according to some timetable. Transfer as incrementals against the previous stored snapshot. Plus you get great data integrity ch…

You can do the same with Ext4

Ext4 has no snapshot feature, do you mean with lvm?

Re: Fire declared in OVH SBG2 datacentre building

#192

Post by ygra »

lrem wrote: post #164
dylan604 wrote: post #62

Earlier quoted context omitted.

If a fire supression kicks in or the fire department shows up with their hoses, would they still say the fire destroyed it or just say destroyed due to fire and water damage? Also, fire suppression system do fail. There was an infamous incident in LA for one of the studios. They built a warehouse to be a tape vault with tapes going back to the 80s. A fire started, but the suppression system failed because there was n…

There's a problem with testing sprinklers: engaging them can be damaging to contents and even structures. So, we're talking about completely emptying the facility, then taking it offline to dry for a time. I've never heard about this being done to anything that was already operational (but I wasn't researching this either).

Isn't something like Halon used in data centers for that reason? That can probably be tested without damaging infrastructure.

Re: Fire declared in OVH SBG2 datacentre building

#193

Post by croon »

AmericanChopper wrote: post #184
tgsovlerkhgsel wrote: post #40

Earlier quoted context omitted.

This problem is usually solved through encryption.

If I were to ask my CISO if I was allowed to bring the production database home, I’m pretty sure his answer wouldn’t be “as long as you encrypt it”.

Well, it's not because the encryption is insecure.

Re: Fire declared in OVH SBG2 datacentre building

#194

Post by CodesInChaos »

opsunit wrote: post #138
pm90 wrote: post #86

Earlier quoted context omitted.

I think the question here, then is of subjective value. AWS may charge more for egress, but that’s not high enough for it to be a concern for most clients. A bigger, independent concern is probably that there should be sufficient redundancy, backups and such that allows for business continuity. (Note again that I’m not saying that all companies make full use of these features, but those that care for such things do.…

An availability zone (AZ) in AWS eu-west-2 was flooded by a fire protection system going off within the last year. It absolutely did affect workloads in that AZ. That shouldn't have had a large impact on their customers since AWS promote and make as trivial as is viable multi-AZ architectures. Put another way: one is guided towards making operational good choices rather than being left to discover them yourself. This…

What surprised me most about today's fire is that their datacenters have so little physical separation. I expected them to be far enough apart to act as separate availability zones.

Re: Fire declared in OVH SBG2 datacentre building

#196

Post by jannes »

pulse7 wrote: post #122
el-salvador wrote: post #107

Earlier quoted context omitted.

Pictures from the fire: https://www.dna.fr/amp/faits-divers-justice/2021/03/10/stras...

Uff... it looks like half of the containers on this picture were on fire...

Are you making a joke about docker containers or am I missing something?

Re: Fire declared in OVH SBG2 datacentre building

#197

Post by optimalsolver »

rgj wrote: post #106
plasma wrote: post #97

“ Update 7:20am Fire is over. Firefighters continue to cool the buildings with the water. We don’t have the access to the site. That is why SBG1, SBG3, SBG4 won’t be restarted today.” https://mobile.twitter.com/olesovhcom/status/136953578757072...

With water, they said.

Liquid cooling.

Re: Fire declared in OVH SBG2 datacentre building

#198

Post by AmericanChopper »

sneak wrote: post #189
AmericanChopper wrote: post #184

Earlier quoted context omitted.

If I were to ask my CISO if I was allowed to bring the production database home, I’m pretty sure his answer wouldn’t be “as long as you encrypt it”.

Once you're big enough to afford a CISO, you're likely big enough to afford office space with decent physical security to serve as a third replicated database site to complement your two datacenters. These solutions are not one-size-fits-all. What works for a small startup isn't appropriate for a 100+ person company.

Yes, I agree. Small companies typically are very bad at security.

Re: Fire declared in OVH SBG2 datacentre building

#199

Post by tomwojcik »

opsunit wrote: post #138
pm90 wrote: post #86

Earlier quoted context omitted.

I think the question here, then is of subjective value. AWS may charge more for egress, but that’s not high enough for it to be a concern for most clients. A bigger, independent concern is probably that there should be sufficient redundancy, backups and such that allows for business continuity. (Note again that I’m not saying that all companies make full use of these features, but those that care for such things do.…

An availability zone (AZ) in AWS eu-west-2 was flooded by a fire protection system going off within the last year. It absolutely did affect workloads in that AZ. That shouldn't have had a large impact on their customers since AWS promote and make as trivial as is viable multi-AZ architectures. Put another way: one is guided towards making operational good choices rather than being left to discover them yourself. This…

Hm, I can't find anything in google about this flooding incident. Can you share some details / source?

Re: Fire declared in OVH SBG2 datacentre building

#200

Post by eb0la »

tothrowaway wrote: post #24
geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traf…

Are you truly paranoid?

If my money and/or job depended on having something running without (or with minimal) disruption I would be as paranoid as you, too.

BTW - Some people call this business recovery plan, not plain paranoia ;-)

Post reply on HN