Earlier quoted context omitted.
Yes, I agree. Small companies typically are very bad at security.
Not in my experience. Worked at some small shops that were lightyears ahead in terms of policy, procedures and attitude compared to places I've worked with 50k+ employees globally.
Fire declared in OVH SBG2 datacentre building
321–330 of 613 posts
Re: Fire declared in OVH SBG2 datacentre building
#322The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…
Re: Fire declared in OVH SBG2 datacentre building
#323The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…
Basically the same (offsite backups), but the details are in the what and how which is subjective... For my purposes I decided that offsite backups should only comprise user data and that all server configuration be 100% scripted with some interactive parts to speed up any customization including recovering backups. I also have my own backup servers rather than using a service, and implement immutable incremental bac…
Re: Fire declared in OVH SBG2 datacentre building
#324The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…
Rolling backups with a month retention to box using rsync.
It creates a network drive to box by default when I boot my desktop.
I have some scripts for putting production db's in test and when I went them locally.
Re: Fire declared in OVH SBG2 datacentre building
#325Unfortunately A lot of people are going to find out the hard way today why AWS/GCP/Big Expensive Cloud is so expensive (Hint: they have redundancy and failover procedures which drive up costs). Keep in mind I’m talking not of “downtime” but of actual data loss which might affect business continuity. This is really tragic. I’m hoping they have some kind of multi regional backup/replication and not just multi zones (al…
“Big cloud” has had fires take out clusters, and somehow they manage to keep it out of the news. In spite of the redundancy and failover procedures, keeping your data centers running when one of the clusters was recently *on fire* is something that is often only possible due to heroic efforts. When I say “heroic efforts”, that’s in contrast to “ordinary error recovery and failover”, which is the way you’d want to han…
Re: Fire declared in OVH SBG2 datacentre building
#326The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…
As an immediate plan, the 2-3 business critical systems are replicating their primary storages to systems in a different datacenter. This allows us to kick off the configuration management in a disaster, and we need something in between 1-4 hours to setup the necessary application servers and middlewares to get critical production running again.
Regarding backups, backups are archived daily to 2 different borg repo hosts on different cloud providers. We could lose an entire hoster to shenanigans and the damage would be limited to ~2 days of data loss at worst. Later this year, we're also considering to export some of these archives to our sister team, so they can place a monthly or weekly backup on tape in a safe in order to have a proper offline backup.
Regarding restores - there are daily automated restore tests for our prod databases, which are then used for a bunch of other tests after anonymization. On top, we've built most database handling on top of the backup/restore infra in order to force us to test these restores during normal business processes.
As I keep saying, installing a database is not hard. Making backups also isn't hard. Ensuring you can restore backups, and ensuring you are not losing backups almost regardless of what happens... that's hard and expensive.
Re: Fire declared in OVH SBG2 datacentre building
#327And this is why the big 3 will continue to dominate. AWS, Microsoft and Google can throw in a lot more money at their phyiscal infrastructure than any other cloud provider. After this sorry episode, I dont think any CTO or CIO of any public company will be able to even consider using the other guys. edit: I am not implying that we put all eggs in one basket with no failover and dr. I am implying the big cos will pay…
Re: Fire declared in OVH SBG2 datacentre building
#328Earlier quoted context omitted.
Basically the same (offsite backups), but the details are in the what and how which is subjective... For my purposes I decided that offsite backups should only comprise user data and that all server configuration be 100% scripted with some interactive parts to speed up any customization including recovering backups. I also have my own backup servers rather than using a service, and implement immutable incremental bac…
This sounds like what I want to do for the new infrastructure I'm setting up in one of OVH's US-based data centers. Are you running on virtual machines or bare metal? What kind of scripting or config management are you using?
They basically install and configure packages using sed or heredocs with a few user prompts here and there for setting up domains etc.
If you are constantly tweaking stuff this might not suit you, but if you know what you need and only occasionally do light changes (which you must ensure the scripts reflect) then this could be an option for you.
It does take some care to write reliable clear bash scripts, and there are some critical choices like `set -e` so that you can walk away and have it hit the end and know that it didn't just error in the middle without you noticing.
Re: Fire declared in OVH SBG2 datacentre building
#329Re: Fire declared in OVH SBG2 datacentre building
#330Earlier quoted context omitted.
Well, it sucks to catch fire and I care for the employees and the firemen, but if their status page is a lie then I have a whole lot less sympathy for the business. That's shady business and they should feel bad. I can appreciate an honest mistake though, like the status page server cron is hosted in the same cluster that caught fire and hence it burnt down and can't update the page anymore.
Is the status page relevant though? At the very least, OVH immediately made a status announcement on their support page and they've been active on Twitter. I don't see anything shady here. From their support page: > The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4. If your production is in Strasbourg, we recommend to activate your Disaster Recovery Plan What more could you…
What's the point of a status page then if it does not show you the status? I don't want to be chasing down twitter handles and support pages during an outage.