21–30 of 119 posts
I admire the goal of reliable long term data storage. I don't need to store tweets or 5TB videos, but my current solution is duplicate DVDs with a bunch of par2 files to hopefully ward off bit rot. I feel like there's some room for improvement.
Earlier quoted context omitted.
You can buy a 2tb drive for $100. $200 and you've got 430 DVD's worth of data redundantly stored. $300 and you've got local redundancy AND offsite backup.
Even if magnetically they don't have 'bit rot', they use bearings where the lubrication can dry up and wear out when they're not turning for long periods of time. You need to keep them spinning on a regular basis, and replace them as they begin to fail.
This is usually a precursor to SMART errors happening in the near future, but unfortunately, it can still result in corrupted replication and corrupted backups; as your backups would be backing up the rotten (corrupt) data.
I've witnessed this happen on both Seagate and WD drives, on systems with ECC memory. I can only suspect this is due to HDD manufacturers wanting to reduce their error rates, and RMA rates: it may happen with the ECC bits in a sector is corrupt, making bitrot undetectable. Instead of giving an error (and being grounds for a RMA replacement), the HDD firmware may choose to return non-integrity-checked data; which would usually be correct but also could be corrupt.
It's why filesystems like ZFS and btrfs are so important.
My rough estimation of this, based on my own experiences and those on r/DataHoarder, suggests 1 hardware sector (4KB for most drives post 2011) will silently corrupt per 10TB year. Such corruption can be detected via checksumming disc systems like ZFS.
Usually, the whole sector is garbage, which is not indicative of cosmic ray bitflips.
External flash memory storage like USB sticks and SD cards fare far worse. In my own experience, silent corruption occurs more like 1 file per device, per 2-3 years; irrespective of the size of memory. I've had USB sticks and SD cards return bogus data without errors, so often. I only know because I checksum everything, otherwise I would have thought the artefacts in my videos or photos came with the source.
If, in 2020, you are not using ZFS or btrfs for long term archival, you are doing something wrong.
ext4, NTFS, APFS, etc may be tried and tested, but they have no checksumming, and that is a problem.
I admire the goal of reliable long term data storage. I don't need to store tweets or 5TB videos, but my current solution is duplicate DVDs with a bunch of par2 files to hopefully ward off bit rot. I feel like there's some room for improvement.
You can always try an affordable online service like Backblaze B2 as one of your options. I haven't checked lately but I think they cost around $5 per terabyte per month. Plus usage, but that would be minimal for an archive.
There are LOTS of failure cases with any cloud provider, especially one with a crazy policy of deleting data in just 45 days.
There is at least 1 reddit post a month about how someone lost data with Backblaze. Their reddit support rep is never able to do anything about it, other than "sorry, we will take on board your feedback".
For comparison, if your Google Drive subscription lapses, Google stops you from uploading but will not delete your data.
Wow, it was really hard to find out what this software does. I finally found a demo at the end of an hour-long talk from 2 years ago: https://youtu.be/PlAU_da_U4s?t=2687 So it seems to have a bunch of scripts to import data into its database from a variety of sources (including cloud services), and provides a search interface to navigate that historical data. And it has a lot of under-the-hood stuff about replication…
yeah like a lot of OSS backup products the docs are grandiose without focusing on ease of operation / feature parity with dropbox
Earlier quoted context omitted.
Even if magnetically they don't have 'bit rot', they use bearings where the lubrication can dry up and wear out when they're not turning for long periods of time. You need to keep them spinning on a regular basis, and replace them as they begin to fail.
HDDs are also prone to silent bitrot, where it will simply return incorrect bytes for a sector, even without any smart errors. (Optical disks also bitrot; but so does HDDs). This is usually a precursor to SMART errors happening in the near future, but unfortunately, it can still result in corrupted replication and corrupted backups; as your backups would be backing up the rotten (corrupt) data. I've witnessed this ha…
Earlier quoted context omitted.
Even if magnetically they don't have 'bit rot', they use bearings where the lubrication can dry up and wear out when they're not turning for long periods of time. You need to keep them spinning on a regular basis, and replace them as they begin to fail.
HDDs are also prone to silent bitrot, where it will simply return incorrect bytes for a sector, even without any smart errors. (Optical disks also bitrot; but so does HDDs). This is usually a precursor to SMART errors happening in the near future, but unfortunately, it can still result in corrupted replication and corrupted backups; as your backups would be backing up the rotten (corrupt) data. I've witnessed this ha…
Earlier quoted context omitted.
You can always try an affordable online service like Backblaze B2 as one of your options. I haven't checked lately but I think they cost around $5 per terabyte per month. Plus usage, but that would be minimal for an archive.
Your credit card may expire or get blocked, and you might be in hospital, or billing alert emails may simply go to your spam folder for 45 days. Your data will be irreversibly deleted by Backblaze. There are LOTS of failure cases with any cloud provider, especially one with a crazy policy of deleting data in just 45 days. There is at least 1 reddit post a month about how someone lost data with Backblaze. Their reddit…
Earlier quoted context omitted.
yeah like a lot of OSS backup products the docs are grandiose without focusing on ease of operation / feature parity with dropbox
no wonder looking at the funding, is it?
Ah, nostalgia :). It seems like with the recent wave of news about social media migrations (reddit, facebook, twitter, twitch, tiktok), people are hopefully starting to get more and more warmed up to the idea of protocolization of their social data. But most of the projects doing it are still just too immature. Solid, Perkeep, Blockstack, etc. just seem like vaporware. Seems like the only serious projects in use are…
To add to your list, there is also Secure Scuttlebutt [1] which has had a decent userbase over the past few years, and Planetary [2] which is a funded iOS client for it.
I think in general they all suffer from the chicken-egg problem and will need some reason for enough people to switch to be able to build a userbase. There isn't really any "novel hook" like tiktok, twitter, whatsapp, instagram, snapchat, etc have had in the past.