According to the official status page, the whole datacenter is still green http://status.ovh.com/vms/index_sbg2.html
Four hours later still green.
Fire declared in OVH SBG2 datacentre building
351–360 of 613 posts
Re: Fire declared in OVH SBG2 datacentre building
#352The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…
My disaster recovery plan: we shall rebuild.
Finally we get to rewrite everything from scratch!
Re: Fire declared in OVH SBG2 datacentre building
#353Earlier quoted context omitted.
I only have a personal server running in Hetzner but it's mirrored onto a tiny local computer at home. They both run postfix + dovecot, so mail is synced via dovecot replication. Data is rsync-ed daily, and everything has ZFS snapshots. MySQL is not set into replication - my home internet breaks often enough to have serious issues, so instead I drop everything every day import a full dump from the main server, and do…
Not saying that you should never do a full mysql dump. Nor that you should not ensure that you can import a full dump. But when you already use ZFS you can do a very speedy full backup with: mysql Transfer the snapshot off-site (and test!). Either as a simple filecopy (the snapshot ensured a consistent database) or a little more advanced with zfs send/receive. This is much quicker and more painless than mysql dump. E…
Do you even need to flush the tables and grab a read lock while taking the ZFS snapshot? My understanding was that since ZFS snapshots are point-in-time consistent, taking a snapshot without flushing tables or grabbing a read lock would be safe; restoring from that snapshot would be like rebooting after losing power.
Re: Fire declared in OVH SBG2 datacentre building
#354And this is why the big 3 will continue to dominate. AWS, Microsoft and Google can throw in a lot more money at their phyiscal infrastructure than any other cloud provider. After this sorry episode, I dont think any CTO or CIO of any public company will be able to even consider using the other guys. edit: I am not implying that we put all eggs in one basket with no failover and dr. I am implying the big cos will pay…
If all of your infrastructure is in one data center, you're on a disaster clock no matter who you choose.
Re: Fire declared in OVH SBG2 datacentre building
#355According to the official status page, the whole datacenter is still green http://status.ovh.com/vms/index_sbg2.html
Also, 3 years ago they had an outage in Strasbourg, and the status page was down apparently as a result of the outage. https://news.ycombinator.com/item?id=15661218 They are not the only ones though. All too common. Well, it's tricky to set this up properly. The only proper way would be to use external infra for the status page.
It's not difficult to make a status page with minimal false negatives. Throw up a server on another host that shows red when it doesn't get a heartbeat. But then instead you end up with false positives. And people will use false positives against you to claim refunds against your SLA.
So nobody chooses to make an honest status page.
Re: Fire declared in OVH SBG2 datacentre building
#356Earlier quoted context omitted.
I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traf…
Are you me, by any chance? :-) I also run our entire production system on one beefy machine at OVH, and replicate to a similar machine at Hetzner. In case of a failure, we just change DNS, which has a 1 hour TTL. We've needed to do an unplanned fail-over only once in over 10 years. And like you, I have an extra replica at the office, because it feels safe having a physical copy of the data literally at hand.
Not done any research into it, but I always thought OVH was supposed to be a very budget VPS service primarily for personal use rather than business. Although thought it was akin to having a Raspberry Pi plugged in at home.
Again, I may be completely wrong but why would you not use AWS/GCP? Even if it's complexity, Amazon have Lightsail, or if it's cost I thought DigitalOcean was one of the only reputable business-grade VPS providers.
I just can't imagine many situations where a VPS would be superior to embracing the cloud and using cloud functions, containers, instances with autoscaling/load balancers etc.
Re: Fire declared in OVH SBG2 datacentre building
#357Earlier quoted context omitted.
Of course it's serious. You are entierly missing the point by quoting the compliance programs followed by AWS whose sole business is being a third party hoster. For most business, what you call sensitive data is customers and orders listing, payment history, inventory if you are dealing in physical goods and HR related files. These are not state secrets. Encryption and a modicum of physical security go a long way. I…
This is quite a significant revision to your previous statement that there’s nothing about a data center that makes it more secure than your house. This attitude that your data isn’t very important, so it’s fine to not be very concerned about it’s security, while not entirely uncommon, is something most organisations try to avoid when choosing vendors. It’s something consumers are generally unconcerned about, until a…
But how many of those splashy breaches ended up being because of the off-site backup copy of the database at the CEO's house?
Re: Fire declared in OVH SBG2 datacentre building
#358I see several people talking about advanced backup systems for businesses.
I do not have a company, I work as a freelancer, I am Brazilian and the current 70 euros of tuition I was paying was already compromising my income, since my local currency is quite devalued against the dollar.
Then imagine the situation. my websites are down, the only backup I have was a copy of the vps that i made in november last year because i was with the intention of creating a server at my house, since it was getting expensive to maintain this server at OVH. It would be unacceptable for a company of this size not to have its servers backed up or to keep them in the same location as the incident, since their networks are all connected. I hope you have a satisfactory and quick solution to this problem.
Re: Fire declared in OVH SBG2 datacentre building
#359For the curious - this is what it looks like when a fire suppression system activates in a (small) server room.
Re: Fire declared in OVH SBG2 datacentre building
#360According to the official status page, the whole datacenter is still green http://status.ovh.com/vms/index_sbg2.html
The weather map is interesting: http://weathermap.ovh.net/
No traffic whatsoever between sbg-g1 and sbg-g2 ant their peers.