Fire declared in OVH SBG2 datacentre building • Hacker News Board

Re: Fire declared in OVH SBG2 datacentre building

#341

Post by Macha » Wed, Mar 10, 2021, 12:24 PM UTC

sofixa wrote: post #182

ericra wrote: post #61

Earlier quoted context omitted.

I opened two tabs to relax tonight: Hacker News and Lichess. This was the top HN thread, and Lichess is having issues because of the fire. I didn't know what OVH was before 10 minutes ago, but this seems really impactful. I hope everyone there is safe and that the immediate disaster gets resolved quickly.

Look them up, they're one of the biggest hosting providers in the world ( especially in France), and due to cheap prices are especially pop-up with smaller scale stuff.

Yeah, they scale down to their kimsufi line which used to have quite powerful dedicated servers for the price of basic VPSes from other providers.

e.g. They have a 4core, 16gb ram server for $22/mo which is 25% of what my preferred provider, Linode, charges.

Now, it comes with older consumer hardware (that one is a sandy bridge i5), and about as much support as the price tag suggests, as well as a dated management interface, but when I used to run a modded minecraft server as a college student, which needed excessive amounts of RAM and could be easily upset by load spikes on other clients, then it was a no-brainer, even if I would expect the modern-ish Xeons Linode uses to win on a core for core basis.

Re: Fire declared in OVH SBG2 datacentre building

#342

Post by Leo_Verto » Wed, Mar 10, 2021, 12:24 PM UTC

londons_explore wrote: post #337

kuschku wrote: post #310

Earlier quoted context omitted.

They're separate buildings, with separate power systems, just standing next to another. Next to SBG2 are also SBG3 and SBG4.

Those buildings weren't quite as far apart as they should have been if a fire in one requires all 4 of the others to be turned off...

Not disagreeing with you but the firefighters probably shut down all power onsite as soon as they arrived.

Re: Fire declared in OVH SBG2 datacentre building

#343

Post by KirillPanov » Wed, Mar 10, 2021, 12:25 PM UTC

The cloud is on fire.

Re: Fire declared in OVH SBG2 datacentre building

#344

Post by [deleted] » Wed, Mar 10, 2021, 12:26 PM UTC

Ploskin wrote: post #313

aetherspawn wrote: post #306

Earlier quoted context omitted.

Well, it sucks to catch fire and I care for the employees and the firemen, but if their status page is a lie then I have a whole lot less sympathy for the business. That's shady business and they should feel bad. I can appreciate an honest mistake though, like the status page server cron is hosted in the same cluster that caught fire and hence it burnt down and can't update the page anymore.

Is the status page relevant though? At the very least, OVH immediately made a status announcement on their support page and they've been active on Twitter. I don't see anything shady here. From their support page: > The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4. If your production is in Strasbourg, we recommend to activate your Disaster Recovery Plan What more could you…

[deleted]

Re: Fire declared in OVH SBG2 datacentre building

#345

Post by AnssiH » Wed, Mar 10, 2021, 12:27 PM UTC

exikyut wrote: post #278

ikiris wrote: post #173

Earlier quoted context omitted.

This was effectively posted on the outages list as well by someone trustworthy. The pictures also look pretty bad from the outside.

Which outages list? Sounds interesting.

I presume they mean this: https://puck.nether.net/mailman/listinfo/outages

Re: Fire declared in OVH SBG2 datacentre building

#346

Post by reasonabl_human » Wed, Mar 10, 2021, 12:31 PM UTC

dividedbyzero wrote: post #291

mike_d wrote: post #263

Earlier quoted context omitted.

Would you humor us with a link to a fully fire proof datacenter?

I think Microsoft had some experimental datacenter containers they submerged in the northern Atlantic for passive cooling, and I believe those were filled with an inert gas as well. I guess that would come very close to an actual fireproof datacenter.

Yep you’re right, learning about this was part of some onboarding training we had to complete...

it was an interesting proof of concept, but finding the right people to maintain the infra with both IT and Scuba skills was a narrow niche to nail down ;)

Re: Fire declared in OVH SBG2 datacentre building

#347

Post by betamaxthetape » Wed, Mar 10, 2021, 12:34 PM UTC

This is certainly a good reminder to have regular backups. I have (had?) a VPS in SBG1, the shipping-container-based data centre in the Strasbourg site, and the latest I know is that out of the 12 containers, 8 are OK but 4 are destroyed [1]. Regardless, I imagine it will be weeks / probably months before even the OK servers can be brought back online.

Naturally, I didn't do regular backups. Most of the data I have from a 2019 backup, but there's a metadata database that I don't have a copy of, and will need to reconstruct from scratch. Thankfully for my case reconstructing will be possible - I know that's not the case for everyone.

Right now I'm feeling pretty stupid, but I only have myself to blame. For their part, OVH have been really good at keeping everyone updated (particularly the Founder and CEO, Octave Klaba).

I believe that when I signed up back in 2017, the Strasbourg location was advertised as one of the cheapest, so I can imagine a lot of people with a ~$4 / month OVH VPS are in the same situation, desperately scrambling to find a backup.

(For those that have a OVH VPS that's down right now, you can find what location it is in by logging onto the OVH control panel.)

[1] https://twitter.com/olesovhcom/status/1369598998441558016

Re: Fire declared in OVH SBG2 datacentre building

#348

Post by christophilus » Wed, Mar 10, 2021, 12:35 PM UTC

manishsharan wrote: post #317

And this is why the big 3 will continue to dominate. AWS, Microsoft and Google can throw in a lot more money at their phyiscal infrastructure than any other cloud provider. After this sorry episode, I dont think any CTO or CIO of any public company will be able to even consider using the other guys. edit: I am not implying that we put all eggs in one basket with no failover and dr. I am implying the big cos will pay…

With Google arbitrarily killing accounts, and with Amazon showing that they’ll do the same if it’s politically expedient, I’m not sure I’d trust the big three, either. It’s a case of “pick your poison”.

Re: Fire declared in OVH SBG2 datacentre building

#349

Post by temp8964 » Wed, Mar 10, 2021, 12:36 PM UTC

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

Do you have access to your control panel when the servers are down?

Re: Fire declared in OVH SBG2 datacentre building

#350

Post by _joel » Wed, Mar 10, 2021, 12:36 PM UTC

AmericanChopper wrote: post #321

_joel wrote: post #277

Earlier quoted context omitted.

Not in my experience. Worked at some small shops that were lightyears ahead in terms of policy, procedures and attitude compared to places I've worked with 50k+ employees globally.

Large organisations tend not to achieve security compliance with overly sophisticated systems of policy and controls. They tend to do it using bureaucracy, which while usually rather effective at implementing the level of control required, will typically leave a lot to be desired in regards to UX and productivity. Small organisations tend to ignore the topic entirely until they encounter a prospective client or regul…

I'm aware, but that's not been my experience. I've been in large places where there's been a lassiez faire attitude because it was "another team's job" and general bikeshedding over smaller features because the bigger picture security wasn't their area or was forced from a dictat from above to use X because they're on the board, whilst X is completely unfit for purpose. There's no pushback. However I've worked at small ISPs where we took security extremely seriously. Appropriate background check and industry policy but moreso the attitude... we wanted to offer customers security because we had pride in our work.