Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

Post reply on HN
Print view View original

531–540 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#531

Post by sandworm101 »

ta988 wrote: post #410
vanviegen wrote: post #155

Earlier quoted context omitted.

Are you me, by any chance? :-) I also run our entire production system on one beefy machine at OVH, and replicate to a similar machine at Hetzner. In case of a failure, we just change DNS, which has a 1 hour TTL. We've needed to do an unplanned fail-over only once in over 10 years. And like you, I have an extra replica at the office, because it feels safe having a physical copy of the data literally at hand.

Same but with a regular offline physical copy (cheap nas). One of my worries is a malicious destruction of the backups if anything worms its way in my network

Which is why "off" is still a great security tool. A copy on a non-powered device, even if that device is attached to the network, is immune to worms. There is something to be said for a NAS solution that requires a physical act to turn on and perform an update.

Re: Fire declared in OVH SBG2 datacentre building

#532

Post by gpm »

namibj wrote: post #505
Twirrim wrote: post #487

Earlier quoted context omitted.

Data centres I used to work in back in the early 2000s had argonite gas dumps in place (prior to argonite, halon used to be popular but is an ozone depleting gas so was phased out) In the case of a fire, it would dump a lot of argonite gas in and consume a large amount of the oxygen in the room, depriving the fire of fuel. It's also safe and leaves minimal clean-up work afterwards, doesn't harm electronics etc. unlik…

Well, yeah, these normal inert gas fire suppression systems don't do a good job if humans can still breathe. The Novec 1230 based ones can actually be sufficiently effective for typical flammability properties you can cheaply adhere to in a datacenter, but even then you iirc would want to add both that and some extra oxygen, because the nitrogen in the air is much more effective at suffocating humans than at suffocat…

> I mean, in principle, if you gave re-breathers to the workers and have some airlocks, you could afford to keep that atmosphere permanently,

At this point I feel like it would be cheaper just to not have workers go there. Fill the place completely full of nitrogen with an onsite nitrogen generator (and only 1atm pressure). Have 100% of regular maintenance and as much irregular maintenance as possible be done by robots. If something happens that requires strength/dexterity beyond the robots (e.g. a heavy object falling over), either have humans go in in some form of scuba gear, or if you can work around it just don't fix it.

Re: Fire declared in OVH SBG2 datacentre building

#534

Post by Scoundreller »

Macha wrote: post #339
Scoundreller wrote: post #171

Earlier quoted context omitted.

and a 2017 (!) article where they were planning to remove 1 and 2 because of power issues! > OVH to Disassemble Container Data Centers after Epic Outage in Europe > “This is probably the worst-case scenario that could have happened to us.” > OVH [...] is planning to shut down and disassemble two of the three data centers on its campus in Strasbourg, France, following a power outage that brought down the entire campus…

It sounds like the plan was to shutdown 1 and 4, the latter of which happened and the former which did not.

It’s confusing as they say they’re shutting down 2 of the 3, but then there’s 4...

Re: Fire declared in OVH SBG2 datacentre building

#535

Post by chrisandchris »

bdz wrote: post #389

Rust (the video game) lost all EU server data w/o restore https://twitter.com/playrust/status/1369611688539009025

Someone forgot the 3-2-1 rule for backups.

I don‘t get why people don‘t do offsite backup today as it‘s basically for free. AWS Glacier basically costs nothing at all.

Re: Fire declared in OVH SBG2 datacentre building

#536

Post by joshuamorton »

fooyc wrote: post #327
manishsharan wrote: post #317

And this is why the big 3 will continue to dominate. AWS, Microsoft and Google can throw in a lot more money at their phyiscal infrastructure than any other cloud provider. After this sorry episode, I dont think any CTO or CIO of any public company will be able to even consider using the other guys. edit: I am not implying that we put all eggs in one basket with no failover and dr. I am implying the big cos will pay…

I could replicate my whole infrastructure on 3 different OVH datacenters, with enough provision to support twice the peak load - it would still be cheaper than a single infrastructure at AWS, and I would get a better uptime than AWS: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

reserved a1.large instances are about half the price of OVH's b2-7 instance. a1.xlarge are still cheaper (and larger). So you get more raw compute per dollar on AWS.

What?

Re: Fire declared in OVH SBG2 datacentre building

#537

Post by kuschku »

ev1 wrote: post #365
londons_explore wrote: post #301

Did OVH claim that SBG1 and SBG2 were isolated failure domains? Despite them just being different rooms in the same building?

They are different physical buildings. OVH does not generally claim anything regarding AZ or distance. SBG1, 2, 3, etc is just denoting the building your server is in - they are not like AWS style AZ or similar, quite literally just building addresses. I have used them for years and I don't believe they've ever said anything like deploy in both SBG1 and SBG2 for safety or availability, because you don't get that choi…

In fact, OVH themselves host e.g. backups of SBG in RBX.

Re: Fire declared in OVH SBG2 datacentre building

#538

Post by kuschku »

joshuamorton wrote: post #536
fooyc wrote: post #327

Earlier quoted context omitted.

I could replicate my whole infrastructure on 3 different OVH datacenters, with enough provision to support twice the peak load - it would still be cheaper than a single infrastructure at AWS, and I would get a better uptime than AWS: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

reserved a1.large instances are about half the price of OVH's b2-7 instance. a1.xlarge are still cheaper (and larger). So you get more raw compute per dollar on AWS. What?

If you need that large machines, and are willing to use reserved instances, you'd go with dedis on OVH instead of VMs. Which is significantly cheaper

Re: Fire declared in OVH SBG2 datacentre building

#539

Post by remram »

gingerlime wrote: post #32
sebmellen wrote: post #18

Earlier quoted context omitted.

Duplicity is your best bet for incremental backups using B2. I use this for my personal server and it works brilliantly.

I thought so too for a long while. Until I was trying to restore something (just to test things), and wasn’t able to... it might have been specific to our GPG or an older version or something... but I decided to switch to restic and am much happier now. Restic has a single binary that takes care of everything. It feels more modern and seems to work really well. Never had any issue restoring from it. Just one data poi…

I've been using Duplicati forever. The fact that it's C# is a bit of a pain (some distros don't have recent Mono), but running it in Docker is easy enough. Being able to check the status of backups and restore files from a web UI is a huge plus, so is the ability to run the same app on all platforms.

I've found duplicity to be a little simplistic and brittle. Purging old backups is also difficult, you basically have to make a full backup (i.e. non-incremental) before you can do that, which increases bandwidth and storage cost.

Restic looks great feature-wise, but still feels like the low-level component you'd use to build a backup system, not a backup system in itself. It's also pre-1.0.

Re: Fire declared in OVH SBG2 datacentre building

#540

Post by rainmaking »

traveler01 wrote: post #441
geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

Most people/companies don't have money to setup those disaster plans. They need you to have a similar server ready to go and also a backuo solution like Amazon S3. I was affected, my personal VPS is safe but down and other VPS I was managing I don't know anything about. I have the backups and right now I'd love for them to just set me up a new VPS so I can restore the backups and restore the services.

Spin it up now and refuse to pay for the old one later.
Post reply on HN