Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

Post reply on HN
Print view View original

361–370 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#361

Post by jakub_g »

kalleboo wrote: post #355
jakub_g wrote: post #305

Earlier quoted context omitted.

Also, 3 years ago they had an outage in Strasbourg, and the status page was down apparently as a result of the outage. https://news.ycombinator.com/item?id=15661218 They are not the only ones though. All too common. Well, it's tricky to set this up properly. The only proper way would be to use external infra for the status page.

It's not difficult to make a status page with minimal false negatives. Throw up a server on another host that shows red when it doesn't get a heartbeat. But then instead you end up with false positives. And people will use false positives against you to claim refunds against your SLA. So nobody chooses to make an honest status page.

Yep. I guess what could be done is a two-tiered status page: automated health check which shows "possible outage, we're investigating" and then a manual update (although some would say it looks lame to say "nah, false positive" which is probably why this setup is rare).

Re: Fire declared in OVH SBG2 datacentre building

#362

Post by CraigJPerry »

klingon78 wrote: post #320
buran77 wrote: post #231

Earlier quoted context omitted.

It's not paranoia if you're right. All of the risks GP is protecting against are things that happen to someone every day, and they should be seen like wearing the seat belt in a car.

I have a reliability and risk avoidance mindset, but I’ve had to stand back because my mental gas tank for trying to keep things going is near empty. I’ve really struggled working with others that either are both ignorant and apathetic about the business’s ability to deal with risk or believe that it’s their job to keep putting duct tape over the duct tape that breaks multiple times a day while users struggle. I like…

This resonates with me. I notice my gas tank rarely depletes because of technology. It doesn’t matter how brain dead the 00’s oracle forms app with absurd unsupported EDI submission excel thinga-ma-bob that requires a modem ... . Making a tech stack safe is a fun challenge.

Apathetic people though, that can be really tough going. It’s just that way “because”. Or my favourite “oh we don’t have permission to change that”, how about we make the case and get permission? _horrified looks_ sometimes followed by pitch forks.

Re: Fire declared in OVH SBG2 datacentre building

#363

Post by ev1 »

Macha wrote: post #341
sofixa wrote: post #182

Earlier quoted context omitted.

Look them up, they're one of the biggest hosting providers in the world ( especially in France), and due to cheap prices are especially pop-up with smaller scale stuff.

Yeah, they scale down to their kimsufi line which used to have quite powerful dedicated servers for the price of basic VPSes from other providers. e.g. They have a 4core, 16gb ram server for $22/mo which is 25% of what my preferred provider, Linode, charges. Now, it comes with older consumer hardware (that one is a sandy bridge i5), and about as much support as the price tag suggests, as well as a dated management in…

Dated? They're probably the only place that isn't $comedy "bare metal cloud" pricing that not only has an API for their $5/m servers but also the panel is a reasonably modern SPA that implements that API and uses OAuth for login

Re: Fire declared in OVH SBG2 datacentre building

#364

Post by Symbiote »

7ewis wrote: post #356
vanviegen wrote: post #155

Earlier quoted context omitted.

Are you me, by any chance? :-) I also run our entire production system on one beefy machine at OVH, and replicate to a similar machine at Hetzner. In case of a failure, we just change DNS, which has a 1 hour TTL. We've needed to do an unplanned fail-over only once in over 10 years. And like you, I have an extra replica at the office, because it feels safe having a physical copy of the data literally at hand.

Not done any research into it, but I always thought OVH was supposed to be a very budget VPS service primarily for personal use rather than business. Although thought it was akin to having a Raspberry Pi plugged in at home. Again, I may be completely wrong but why would you not use AWS/GCP? Even if it's complexity, Amazon have Lightsail, or if it's cost I thought DigitalOcean was one of the only reputable business-gr…

OVH is a European equivalent to Digital Ocean.

It has twice the revenue, and is the third largest hosting provider in the world.

Re: Fire declared in OVH SBG2 datacentre building

#365

Post by ev1 »

londons_explore wrote: post #301

Did OVH claim that SBG1 and SBG2 were isolated failure domains? Despite them just being different rooms in the same building?

They are different physical buildings. OVH does not generally claim anything regarding AZ or distance. SBG1, 2, 3, etc is just denoting the building your server is in - they are not like AWS style AZ or similar, quite literally just building addresses.

I have used them for years and I don't believe they've ever said anything like deploy in both SBG1 and SBG2 for safety or availability, because you don't get that choice.

When you provision a machine (eg via API) they tell you "SBG 5 min, LON 5 min, BHS 72h" and you pick SBG and get assigned first-available. There is no "I want to be in SBG4" generally.

Re: Fire declared in OVH SBG2 datacentre building

#366

Post by SergeAx »

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

I don't have to "backup servers" for a long time now. I have an Ansible playbook to deploy and orchestrate services, which, in turn, are mostly dockerized. So my recovery plan is to turn on "sorry, maintenance" banner via CDN, spin up a bunch of new VPSes, run Ansible scenario for deployment and restore database from hidden replica or latest dump.

Re: Fire declared in OVH SBG2 datacentre building

#367

Post by dboreham »

SergeAx wrote: post #366
geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

I don't have to "backup servers" for a long time now. I have an Ansible playbook to deploy and orchestrate services, which, in turn, are mostly dockerized. So my recovery plan is to turn on "sorry, maintenance" banner via CDN, spin up a bunch of new VPSes, run Ansible scenario for deployment and restore database from hidden replica or latest dump.

> restore database from hidden replica or latest dump

You do have backup servers.

Re: Fire declared in OVH SBG2 datacentre building

#368

Post by [deleted] »

dboreham wrote: post #367
SergeAx wrote: post #366

Earlier quoted context omitted.

I don't have to "backup servers" for a long time now. I have an Ansible playbook to deploy and orchestrate services, which, in turn, are mostly dockerized. So my recovery plan is to turn on "sorry, maintenance" banner via CDN, spin up a bunch of new VPSes, run Ansible scenario for deployment and restore database from hidden replica or latest dump.

> restore database from hidden replica or latest dump You do have backup servers.

[deleted]

Re: Fire declared in OVH SBG2 datacentre building

#369

Post by MegaThorx »

r4ibOm wrote: post #358

I see several people talking about advanced backup systems for businesses. I do not have a company, I work as a freelancer, I am Brazilian and the current 70 euros of tuition I was paying was already compromising my income, since my local currency is quite devalued against the dollar. Then imagine the situation. my websites are down, the only backup I have was a copy of the vps that i made in november last year becau…

I had my server in SBG2 and sadly the backup failed since the end of january. Yep it is my mistake for not checking the backups. Now I lost about 1 month of data.

The only good thing is that my backup was offsite.

Does OVH offer automatic snapshots for VPS? I know Hetzner does it for 20% addidation of the cost of the server. If they do the next question would be if they are destroyed too?

Re: Fire declared in OVH SBG2 datacentre building

#370

Post by olivierduval »

7ewis wrote: post #356
vanviegen wrote: post #155

Earlier quoted context omitted.

Are you me, by any chance? :-) I also run our entire production system on one beefy machine at OVH, and replicate to a similar machine at Hetzner. In case of a failure, we just change DNS, which has a 1 hour TTL. We've needed to do an unplanned fail-over only once in over 10 years. And like you, I have an extra replica at the office, because it feels safe having a physical copy of the data literally at hand.

Not done any research into it, but I always thought OVH was supposed to be a very budget VPS service primarily for personal use rather than business. Although thought it was akin to having a Raspberry Pi plugged in at home. Again, I may be completely wrong but why would you not use AWS/GCP? Even if it's complexity, Amazon have Lightsail, or if it's cost I thought DigitalOcean was one of the only reputable business-gr…

OVH STARTED as a budget VPS service some 20 years ago... but they grew a lot since 6-7 years, adding more "cloud" services and capabilities, even not on par with the main players...

Why not use AWS/GCP? From my personal point of view: as a French citizen, I'm more and more convinced that I can't completly trust the (US) big boys for my own safety. Trump showed that "US interest" is far more important than "customer interest" or even "ally interest". And moreover, Google is showing quite regurlaly that it's not a reliable business partner (AWS look better for this).

Post reply on HN