Keep Your Stuff, for Life • Hacker News Board

Re: Keep Your Stuff, for Life

#31

Post by roland-s » Tue, Jun 30, 2020, 7:12 AM UTC

harikb wrote: post #26

dannyw wrote: post #23

Earlier quoted context omitted.

HDDs are also prone to silent bitrot, where it will simply return incorrect bytes for a sector, even without any smart errors. (Optical disks also bitrot; but so does HDDs). This is usually a precursor to SMART errors happening in the near future, but unfortunately, it can still result in corrupted replication and corrupted backups; as your backups would be backing up the rotten (corrupt) data. I've witnessed this ha…

Do proper use of ZFS also require ECC memory?

100% yes. With non-ECC you will always have bad RAM bits eventually. With ZFS this is especially bad because it can corrupt your checksums or your ZFS metadata, which means either silently corrupting your data, or corrupting ZFS itself and losing your entire zpool (akin to losing a RAID array).

Re: Keep Your Stuff, for Life

#32

Post by teekert » Tue, Jun 30, 2020, 7:19 AM UTC

CGamesPlay wrote: post #9

Wow, it was really hard to find out what this software does. I finally found a demo at the end of an hour-long talk from 2 years ago: https://youtu.be/PlAU_da_U4s?t=2687 So it seems to have a bunch of scripts to import data into its database from a variety of sources (including cloud services), and provides a search interface to navigate that historical data. And it has a lot of under-the-hood stuff about replication…

This comment represents the value of HackerNews comments very well :) Thank you.

Re: Keep Your Stuff, for Life

#33

Post by kstenerud » Tue, Jun 30, 2020, 7:20 AM UTC

na85 wrote: post #5

I admire the goal of reliable long term data storage. I don't need to store tweets or 5TB videos, but my current solution is duplicate DVDs with a bunch of par2 files to hopefully ward off bit rot. I feel like there's some room for improvement.

Hook up ZRaid SSDs on a low powered machine (like raspberry pi). Set up regular ZFS scrubbing, and connect that to a monitoring service.

You're not going to prevent silent bitrot no matter what modern technology you use, so take a proactive approach instead to prevent data loss.

Re: Keep Your Stuff, for Life

#34

Post by reallydontask » Tue, Jun 30, 2020, 7:21 AM UTC

TuringTest wrote: post #29

mro_name wrote: post #25

Earlier quoted context omitted.

no wonder looking at the funding, is it?

You don't need funding for focusing on ease of operation, it's a mindset. Either you're user-centered, or you're not.

>Either you're user-centered, or you're not.

It really depends on the audience for the product, targeting regular users does require more thought on the UX, etc which almost invariably means more funding.

Re: Keep Your Stuff, for Life

#35

Post by HenryBemis » Tue, Jun 30, 2020, 7:39 AM UTC

denkmoon wrote: post #13

na85 wrote: post #5

I admire the goal of reliable long term data storage. I don't need to store tweets or 5TB videos, but my current solution is duplicate DVDs with a bunch of par2 files to hopefully ward off bit rot. I feel like there's some room for improvement.

You can buy a 2tb drive for $100. $200 and you've got 430 DVD's worth of data redundantly stored. $300 and you've got local redundancy AND offsite backup.

My setup (laptop w8.1pro): external 4TB disk, assigned to letter L (for Library). I got Acronis running once per month, dumping a 70-80GB .tib file on L. L also has a backup folder with everything I got (setup files, books, photos, every audiobook/video I need such as trainings, etc). The whole backup is ~2TB.

Now get Carbonite (not affiliated, I just like the infinite space backup), and get it to backup your key laptop folders (Docs, Images, Desktop, etc) and your L-drive. I don't remember how much it costs ($6-10?/mo), but I have stopped worrying since then. I got a monthly tib file for my system and an "instant" backup for everything else. So even if my laptop is stolen I can set up a new laptop (the .tib may be useless but I can open it and see what s/w I had and I can take the config files/folders to move to my new system).

I don't remember how much the disk was but it didn't hurt my wallet, and the ~$100 (?) per year on Carbonite (had CrashPlan) definitely doesn't hurt my wallet.

Re: Keep Your Stuff, for Life

#36

Post by pwg » Tue, Jun 30, 2020, 7:48 AM UTC

na85 wrote: post #5

I admire the goal of reliable long term data storage. I don't need to store tweets or 5TB videos, but my current solution is duplicate DVDs with a bunch of par2 files to hopefully ward off bit rot. I feel like there's some room for improvement.

If you continue your DVD method you may also want to look into "dvdisaster" https://en.wikipedia.org/wiki/Dvdisaster

Re: Keep Your Stuff, for Life

#37

Post by drwolff » Tue, Jun 30, 2020, 7:57 AM UTC

CGamesPlay wrote: post #9

Wow, it was really hard to find out what this software does. I finally found a demo at the end of an hour-long talk from 2 years ago: https://youtu.be/PlAU_da_U4s?t=2687 So it seems to have a bunch of scripts to import data into its database from a variety of sources (including cloud services), and provides a search interface to navigate that historical data. And it has a lot of under-the-hood stuff about replication…

Same video is posted at the bottom of their website.

Re: Keep Your Stuff, for Life

#38

Post by pseudalopex » Tue, Jun 30, 2020, 8:11 AM UTC

harikb wrote: post #26

dannyw wrote: post #23

Earlier quoted context omitted.

HDDs are also prone to silent bitrot, where it will simply return incorrect bytes for a sector, even without any smart errors. (Optical disks also bitrot; but so does HDDs). This is usually a precursor to SMART errors happening in the near future, but unfortunately, it can still result in corrupted replication and corrupted backups; as your backups would be backing up the rotten (corrupt) data. I've witnessed this ha…

Do proper use of ZFS also require ECC memory?

ZFS protects you from disk errors. ECC protects you from memory errors. Using one or the other is safer than using neither. Using both together is even safer.

Re: Keep Your Stuff, for Life

#39

Post by kurlberg » Tue, Jun 30, 2020, 8:15 AM UTC

roland-s wrote: post #31

harikb wrote: post #26

Earlier quoted context omitted.

Do proper use of ZFS also require ECC memory?

100% yes. With non-ECC you will always have bad RAM bits eventually. With ZFS this is especially bad because it can corrupt your checksums or your ZFS metadata, which means either silently corrupting your data, or corrupting ZFS itself and losing your entire zpool (akin to losing a RAID array).

Maybe not: that ZFS needs ECC is "common wisdom", but the disaster scenario appears not so likely. See

https://news.ycombinator.com/item?id=14207520

https://news.ycombinator.com/item?id=8293025

Re: Keep Your Stuff, for Life

#40

Post by dabeeeenster » Tue, Jun 30, 2020, 8:30 AM UTC

ahupp wrote: post #22

It sounds like this preserves your data from lots of different services. This is something I need! But I couldn't figure out what it actually supports. Suggestion: the very first paragraph should describe the specific inputs it can handle.

spent a minute looking and couldnt find the list...