Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keep Your Stuff, for Life

perkeep.org

Post reply on HN
Print view View original

41–50 of 119 posts

Re: Keep Your Stuff, for Life

#41

Post by solarkraft »

Perkeep is super interesting. I have been looking at it as a media database (photos, movies and such, powerful tag-based search, all downloaded on demand), but some things regarding that really hurt:

- The data store messes with my files (yes, there's a FUSE mount, but eh, having to adopt a special data store always makes me feel weird since it usually comes with performance and compatiiblity implications, there are also many other block-chopping data stores, for example in IPFS).

- The last time I checked there was no way to delete something. This is okay for Tweets I guess, but if I commit a 3h video I later realize to just be too large or a photo I end up not really wanting around - well, oops.

I have huge respect for Brad Fitzpatrick in general, of course, and especially for creating this. Recent velocity has seemilgly been relatively low, however: https://news.ycombinator.com/item?id=22161812

Re: Keep Your Stuff, for Life

#42

Post by BiteCode_dev »

TuringTest wrote: post #29
mro_name wrote: post #25

Earlier quoted context omitted.

no wonder looking at the funding, is it?

You don't need funding for focusing on ease of operation, it's a mindset. Either you're user-centered, or you're not.

Why do you think cmd line tools are popular ? Because they are easier to do than a GUI, not because all geeks think the terminal is the only xp that matters.

Re: Keep Your Stuff, for Life

#43

Post by atq2119 »

BiteCode_dev wrote: post #42
TuringTest wrote: post #29

Earlier quoted context omitted.

You don't need funding for focusing on ease of operation, it's a mindset. Either you're user-centered, or you're not.

Why do you think cmd line tools are popular ? Because they are easier to do than a GUI, not because all geeks think the terminal is the only xp that matters.

That seems... offtopic? You can have command line tools that are easy to use, with documentation focused on user-friendliness, and you can have command line tools where figuring out what they even do in the first place takes a lot of effort. And the same is true for GUIs, for what it's worth.

Re: Keep Your Stuff, for Life

#44

Post by ZenPsycho »

CGamesPlay wrote: post #9

Wow, it was really hard to find out what this software does. I finally found a demo at the end of an hour-long talk from 2 years ago: https://youtu.be/PlAU_da_U4s?t=2687 So it seems to have a bunch of scripts to import data into its database from a variety of sources (including cloud services), and provides a search interface to navigate that historical data. And it has a lot of under-the-hood stuff about replication…

it’s a distributed personal content addressed blob object store, with a variety of configurable storage back ends, plus a web interface to make that kind of usable to a human for human things.

it’s like git but for all your stuff.

why would you want to use it? you probably wouldn’t, quite yet. but it’s an interesting attempt at doing something a little more sophisticated than a plain file system.

Re: Keep Your Stuff, for Life

#45

Post by [deleted] »

BiteCode_dev wrote: post #42
TuringTest wrote: post #29

Earlier quoted context omitted.

You don't need funding for focusing on ease of operation, it's a mindset. Either you're user-centered, or you're not.

Why do you think cmd line tools are popular ? Because they are easier to do than a GUI, not because all geeks think the terminal is the only xp that matters.

[deleted]

Re: Keep Your Stuff, for Life

#46

Post by mnw21cam »

kstenerud wrote: post #33
na85 wrote: post #5

I admire the goal of reliable long term data storage. I don't need to store tweets or 5TB videos, but my current solution is duplicate DVDs with a bunch of par2 files to hopefully ward off bit rot. I feel like there's some room for improvement.

Hook up ZRaid SSDs on a low powered machine (like raspberry pi). Set up regular ZFS scrubbing, and connect that to a monitoring service. You're not going to prevent silent bitrot no matter what modern technology you use, so take a proactive approach instead to prevent data loss.

Agreed. I used to back up onto DVDs, creating a set of (say) 12 DVDs with an extra couple generated by par2, to take care of the case where some of the DVDs just straight aren't readable.

However, I found that I had a lot of data to back up, and it was actually cheaper and less tedious to get 4TB USB hard drives for ~£100 each, and plug them into an old defunct EeePC901 (with the added advantage that if the power goes out it has a battery).

My main PC has an SSH private key that lets it access a restricted shell on the EeePC that only allows it to give it files to store. That way, if a hacker breaks into my internet-facing machine, all they can do to my backups is fill the disc up, not delete or access anything. I have a process on the EeePC that regularly scrubs the par2 files, and the hard drives (I have two so far) are formatted with BTRFS, so given all the data is regularly read by the scrub process, that should notice any drive failures. My main PC uses ZFS, so I have safety in variety.

I also have an off-site backup stored on an encrypted USB hard drive in my locked locker at work, which isn't updated as regularly. My internet connection is slow, so I use the rsync --only-write-batch trick, and then carry the large update file to the backup on my laptop.

What could possibly go wrong?

Re: Keep Your Stuff, for Life

#48

Post by eexan »

I tried perkeep a while ago. While the ideas are cool, the implementation is meh:

I added a single 2.7G ubuntu iso - it took 5 minutes to ingest it (on a tmpfs!), and turned it into 45k(!) little chunks, wtf is up with that? At this rate indexing my multiple terabytes of data is going to take it days and I don't even want to think how much seek time it's going to need to store its repo on a spinning HDD.

Re: Keep Your Stuff, for Life

#49

Post by mnw21cam »

dannyw wrote: post #23
paranoidrobot wrote: post #17

Earlier quoted context omitted.

Even if magnetically they don't have 'bit rot', they use bearings where the lubrication can dry up and wear out when they're not turning for long periods of time. You need to keep them spinning on a regular basis, and replace them as they begin to fail.

HDDs are also prone to silent bitrot, where it will simply return incorrect bytes for a sector, even without any smart errors. (Optical disks also bitrot; but so does HDDs). This is usually a precursor to SMART errors happening in the near future, but unfortunately, it can still result in corrupted replication and corrupted backups; as your backups would be backing up the rotten (corrupt) data. I've witnessed this ha…

Interestingly, on my home ZFS raidz with 3 4TB hard drives, I have had to replace a drive a couple of times because ZFS scrub was reporting silent corruption. They were consumer-grade SATA drives.

However, at work, I have backed up ~200TB of data to a large server with RAID-6 and ext4, storing the backups as large .tar files with par2 checksums and recovery data, and regularly scrubbing the par2 data. I have yet to see any corruption whatsoever. These are enterprise-grade hard drives. This is the strongest evidence I have yet seen that the enterprise-grade drives are actually better than the consumer-grade ones, rather than just being re-badged.

Re: Keep Your Stuff, for Life

#50

Post by kasabali »

paranoidrobot wrote: post #17
denkmoon wrote: post #13

Earlier quoted context omitted.

You can buy a 2tb drive for $100. $200 and you've got 430 DVD's worth of data redundantly stored. $300 and you've got local redundancy AND offsite backup.

Even if magnetically they don't have 'bit rot', they use bearings where the lubrication can dry up and wear out when they're not turning for long periods of time. You need to keep them spinning on a regular basis, and replace them as they begin to fail.

I'm no expert on the issue so correct me if I'm wrong, but I've heard modern HDDs use fluid bearings and isn't susceptible to drying up.
Post reply on HN