Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

Post reply on HN
Print view View original

371–380 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#371

Post by sigotirandolas »

keraf wrote: post #272
sgeisler wrote: post #249

Earlier quoted context omitted.

It seems to be a static site, which seems reasonable since it aggregates a lot of data and might encounter high load when something goes wrong, so generating it live without caching is not viable. So maybe the server that normally updates it is down too (not that this would be a good excuse)?

10+ hours cache on a status page doesn't look like real-time monitoring to me. I think this is probably linked to a manual reporting system and they got bigger fish to fry at the moment than updating this status page.

Counterpoint: There's a constantly updating timestamp at the top of the page that suggests it's automated and real time.

Re: Fire declared in OVH SBG2 datacentre building

#372

Post by Symbiote »

r4ibOm wrote: post #358

I see several people talking about advanced backup systems for businesses. I do not have a company, I work as a freelancer, I am Brazilian and the current 70 euros of tuition I was paying was already compromising my income, since my local currency is quite devalued against the dollar. Then imagine the situation. my websites are down, the only backup I have was a copy of the vps that i made in november last year becau…

If OVH backed up everything, the cost of the service would be double.

Many customers don't need a backup, so it's up to each customer to arrange their own backups — perhaps with tools and services provided by the hosting company, or their own solution.

Running a company with no backup (for cost or any other reason) is very risky, as some people will have found out today.

Re: Fire declared in OVH SBG2 datacentre building

#374

Post by ev1 »

7ewis wrote: post #356
vanviegen wrote: post #155

Earlier quoted context omitted.

Are you me, by any chance? :-) I also run our entire production system on one beefy machine at OVH, and replicate to a similar machine at Hetzner. In case of a failure, we just change DNS, which has a 1 hour TTL. We've needed to do an unplanned fail-over only once in over 10 years. And like you, I have an extra replica at the office, because it feels safe having a physical copy of the data literally at hand.

Not done any research into it, but I always thought OVH was supposed to be a very budget VPS service primarily for personal use rather than business. Although thought it was akin to having a Raspberry Pi plugged in at home. Again, I may be completely wrong but why would you not use AWS/GCP? Even if it's complexity, Amazon have Lightsail, or if it's cost I thought DigitalOcean was one of the only reputable business-gr…

OVH is one of the largest providers in the world. They run a sub brand for personal use (bare metal for $5/m, hardware replacements in 30 min or less usually).

..and they do support all of those things you just listed, not just API-backed bare metal.

Re: Fire declared in OVH SBG2 datacentre building

#375

Post by veeti »

7ewis wrote: post #356
vanviegen wrote: post #155

Earlier quoted context omitted.

Are you me, by any chance? :-) I also run our entire production system on one beefy machine at OVH, and replicate to a similar machine at Hetzner. In case of a failure, we just change DNS, which has a 1 hour TTL. We've needed to do an unplanned fail-over only once in over 10 years. And like you, I have an extra replica at the office, because it feels safe having a physical copy of the data literally at hand.

Not done any research into it, but I always thought OVH was supposed to be a very budget VPS service primarily for personal use rather than business. Although thought it was akin to having a Raspberry Pi plugged in at home. Again, I may be completely wrong but why would you not use AWS/GCP? Even if it's complexity, Amazon have Lightsail, or if it's cost I thought DigitalOcean was one of the only reputable business-gr…

AWS is a total and utter ripoff compared to the price/performance, DDoS protection & unmetered bandwidth provided by OVH.

Re: Fire declared in OVH SBG2 datacentre building

#376

Post by potemkinhr »

reasonabl_human wrote: post #346
dividedbyzero wrote: post #291

Earlier quoted context omitted.

I think Microsoft had some experimental datacenter containers they submerged in the northern Atlantic for passive cooling, and I believe those were filled with an inert gas as well. I guess that would come very close to an actual fireproof datacenter.

Yep you’re right, learning about this was part of some onboarding training we had to complete... it was an interesting proof of concept, but finding the right people to maintain the infra with both IT and Scuba skills was a narrow niche to nail down ;)

I don't opening it at any point before decomissioning it completely is even an afterthought with that. They just write off any failures and roll with it as long as it's viable.

Re: Fire declared in OVH SBG2 datacentre building

#377

Post by buran77 »

klingon78 wrote: post #320
buran77 wrote: post #231

Earlier quoted context omitted.

It's not paranoia if you're right. All of the risks GP is protecting against are things that happen to someone every day, and they should be seen like wearing the seat belt in a car.

I have a reliability and risk avoidance mindset, but I’ve had to stand back because my mental gas tank for trying to keep things going is near empty. I’ve really struggled working with others that either are both ignorant and apathetic about the business’s ability to deal with risk or believe that it’s their job to keep putting duct tape over the duct tape that breaks multiple times a day while users struggle. I like…

Reliability is there to keep your things running smoothly during normal operations. Backups are there for when you reach the end of your reliability rope. Neither is really a good replacement for the other. The most reliable systems will still fail eventually, and the best of backups can't run your day to day operations.

At the end of the day you have a budget (of any kind) and a list of priorities on which to spend it. It's up to you or your management to set a reasonable budget, and to set the right priorities. If they refuse, leave or you'll just burn the candle at both ends and just fade out.

Re: Fire declared in OVH SBG2 datacentre building

#378

Post by omnimus »

7ewis wrote: post #356
vanviegen wrote: post #155

Earlier quoted context omitted.

Are you me, by any chance? :-) I also run our entire production system on one beefy machine at OVH, and replicate to a similar machine at Hetzner. In case of a failure, we just change DNS, which has a 1 hour TTL. We've needed to do an unplanned fail-over only once in over 10 years. And like you, I have an extra replica at the office, because it feels safe having a physical copy of the data literally at hand.

Not done any research into it, but I always thought OVH was supposed to be a very budget VPS service primarily for personal use rather than business. Although thought it was akin to having a Raspberry Pi plugged in at home. Again, I may be completely wrong but why would you not use AWS/GCP? Even if it's complexity, Amazon have Lightsail, or if it's cost I thought DigitalOcean was one of the only reputable business-gr…

You cant imagine it yet big chunk of the independent internet runs on small vps servers. There isnt much difference between DO and OVH, Hetzner, Vultr, Linode... not sure why DO would be better. I mean its US company doing marketing right. Thats the difference. Plus ovh/hetzner have only EU locations.

I think small bussinesses like smaller simple providers instead of bigclouds. Its different philosophy if you are afraid of extreme centralisation of internet it makes sense.

Re: Fire declared in OVH SBG2 datacentre building

#379

Post by jt2190 »

klingon78 wrote: post #320
buran77 wrote: post #231

Earlier quoted context omitted.

It's not paranoia if you're right. All of the risks GP is protecting against are things that happen to someone every day, and they should be seen like wearing the seat belt in a car.

I have a reliability and risk avoidance mindset, but I’ve had to stand back because my mental gas tank for trying to keep things going is near empty. I’ve really struggled working with others that either are both ignorant and apathetic about the business’s ability to deal with risk or believe that it’s their job to keep putting duct tape over the duct tape that breaks multiple times a day while users struggle. I like…

Consider:

> ... [F]inance is fundamentally about moving money and risk through a network. [1]

Your employer has taken on many, many risks as part of their enterprise. If every risk is addressed the company likely can’t operate profitably. In this context, your business needs to identify every risk, weigh the likelihood and the potential impact, decide whether to address or accept the risk, and finally, if they decide to address the risk, whether to address it in-house our outsource it.

You’ve identified a risk that is currently being “accepted” by your employer, one that you’d like to address in-house. Perhaps they’ve taken on the risk unintentionally, out of ignorance.

As a professional the best I can do is to make sure that the business isn’t ignorant about the risk they’ve taken on. If the risk is too great I might even leave. Beyond that I accept that life is full of risks.

[1] Gary Gensler, “Blockchain and money”, Introduction https://ocw.mit.edu/courses/sloan-school-of-management/15-s1...

Re: Fire declared in OVH SBG2 datacentre building

#380

Post by sebastianconcpt »

dave_sullivan wrote: post #300
geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

I am literally in SBG2 so that has been fun. Turns out, our disaster recovery plan is pretty good. Datacenter burned down and I still was up 4 hours later in another data center with zero data loss. Good times.

And how's SBG1 doing?
Post reply on HN