Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

581–590 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#581
post #67

A status update on the OVH tracker for a different datacenter (LIM-1 / Limburg) says "We are going to intervene in the rack to replace a large number of power supply cables that could have an insulation defect." [0][1] The same type of issue is "planned" in BHS [3] and GRA [2]. Eerie timing: do they possibly suspect some bad cables? [0]: http://travaux.ovh.net/?do=details&id=49016 [1]: http://travaux.ovh.net/?do=deta…

I had mine restarted this morning in BHS. Only one of 3 servers...

Re: Fire declared in OVH SBG2 datacentre building

#582
post #557

Earlier quoted context omitted.

> They did not have a fire suppression system I find it very hard to believe that that would pass code anywhere in the US/EU or most of the world. They may not have had sprinklers but that doesn't mean there isn't fire suppression.

According to a forum post discussing the building of said DC, with photos and such, there's no visible fire suppression system: https://lafibre.info/ovh-datacenter/ovh-et-la-protection-inc...

The floors are plywood and it's built with wood frame construction.

I retract all my prior surprise that this place burned down.

Re: Fire declared in OVH SBG2 datacentre building

#583
post #567
post #153

Earlier quoted context omitted.

Yup, I've never heard of a fire taking out a Big Cloud DC. They actually know what they're doing and don't put server racks in shipping containers stacked on top of each other. If you want quality in life, sometimes you have to pay for it. Personally I'll continue to use these third world cloud providers. But I like to live on the edge.

Apple fire: https://www.datacenterdynamics.com/en/news/fire-rages-throug... Google fire: https://www.google.nl/amp/s/gigazine.net/amp/en/20060313_goo... AWS fire: https://money.cnn.com/2015/01/09/technology/amazon-data-cent...

> Apple fire

Their solar panels on the roof caught on fire. Presumably there was no actual disruption in service computing-wise.

> Google fire

2006, still a young company at that point. Also this was way before GCP where they were responsible for other people's services.

> AWS fire

Data center caught on fire while it was under construction. Drunk construction workers messed up, not AWS. Presumably no computers were even inside yet. This is completely different from OVH completely burning down while all their customers' data went up in flames.

If these are the best examples of Big Cloud messing up, this is quite the endorsement.

It's funny how the OVH CEO tells everyone to go activate their DRPs while the company didn't have enough foresight to install fire suppression systems.

Re: Fire declared in OVH SBG2 datacentre building

#584
post #146

According to the official status page, the whole datacenter is still green http://status.ovh.com/vms/index_sbg2.html

I feel like there should be place to report infrastructure suppliers with misleading status pages, some kind of crowdsourced database. Without this information, you only find out that they are misleading when something goes very wrong. At best you might be missing out on some SLA refunds, but at worst it could be disasterous for a business. I've been on the wrong side of a update-by-hand status system from a hosting…

Would a blatantly false status page meet the criteria for false advertising?

My gut says yes.

Re: Fire declared in OVH SBG2 datacentre building

#585

Earlier quoted context omitted.

I'm not the OP, but I'd imagine it is the combination of no support line and algorithmic suspension of business accounts. It is a relevant risk.

Yeah, I was thinking about all the horror stories that can be found on this site. As a customer (or maybe an "involontary data provider"), I do as much as I can to avoid Google to be my SPOF, not technically (it's really technically reliable) but on the business side. I had to setup my own mail server just to avoid any risk of google-ban for example... just in case. I won't use Google authentificator for the same rea…

Nope. I was at a company with a $1M dedicated spend contract w/ GCP and what that got us was support through a VAR. It then became the VAR’s job to file support tickets that took two weeks to get the response “oh well that’s now how we do it at Google. Have you read these docs you already said you read and can you send logs you already sent?” instead of my job to do that.

Re: Fire declared in OVH SBG2 datacentre building

#586
post #578
post #447

Earlier quoted context omitted.

OVH has new servers. Their sub-brand soyoustart has older servers (that are still perfectly fine), roughly E3 Xeon/16-32GB/3x2TB to 4x2TB for $40/m ex vat. Their other sub brand kimsufi for personal servers has Atom low-power bare metal with 2TB HDD (in reality it is advertised 500GB/1TB, but they don't really have any of those in stock left, if your drive fails they replace it with a 2T - so far this has been my exp…

Just as a side note, the name "kimsufi" comes from the French "qui me suffit", which roughly translate to "enough for me".

Ha! You can see they localise it too https://www.ovh.com/world/support/terms-and-conditions/

Like kimsufi equivalent brand "isgenug"

Re: Fire declared in OVH SBG2 datacentre building

#587

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

Never thought i'd read the header

> Printer flammability

Re: Fire declared in OVH SBG2 datacentre building

#588
post #39

Earlier quoted context omitted.

I have three servers (1 OVH - different location, 2 DO). The only thing I backup is the DB, which is synced daily to S3. There's a rule to automatically delete files after 30 days to handle GDPR and stop the bucket and costs spiralling out of control. Everything is managed with Ansible and Terraform (on DO side), so I could probably get everything back up and running in less than an hour if needed.

> probably That makes it sound like you didn't try/practice. I imagine that in a real-life scenario things will be a little more painful than in one's imagination.

I haven't ever deleted everything and timed how long I could get it up and running again, but I have tested it works by spinning up new machines and moving everything over to there (it was easier than running "sudo apt-get dist-upgrade").

Re: Fire declared in OVH SBG2 datacentre building

#589
post #192
post #164

Earlier quoted context omitted.

There's a problem with testing sprinklers: engaging them can be damaging to contents and even structures. So, we're talking about completely emptying the facility, then taking it offline to dry for a time. I've never heard about this being done to anything that was already operational (but I wasn't researching this either).

Isn't something like Halon used in data centers for that reason? That can probably be tested without damaging infrastructure.

Good point. But, apart from cost of the gas, mentioned by sibling comment, that also isn't disruption-free. Gas-based systems are by definition (they displace oxygen) dangerous to humans. But yeah, not being able to run maintenance is orders of magnitude less problematic.

Re: Fire declared in OVH SBG2 datacentre building

#590
post #451

Earlier quoted context omitted.

As a quick interlude, since this may be confusing to non-US readers: bankruptcy in the United States in the context of business usually refers to two concepts, whereas in many other countries it refers to just one. There are two types of bankruptcies in the US used most often by insolvent businesses: Chapter 7, and Chapter 11. A Chapter 7 bankruptcy is what most people in other countries think of when they hear "bank…

This helps a lot, thanks. I think most people international would assume bankruptcy = game over.

Yes, I for one, was confused.

I wondered why you should plan an event which will "destroy" your company anyways.

Post reply on HN