Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Nearly Lost All of My Data

kevq.uk

Post reply on HN
Print view View original

191–200 of 262 posts

Re: I Nearly Lost All of My Data

#191

Post by cyphar »

KingMachiavelli wrote: post #90
_bxg1 wrote: post #74

I just don't understand rolling your own backup drives in 2018 unless you're up to something nefarious. I rest peacefully knowing my data will never be lost by Dropbox, and the immediate backup means I don't have to mess with slow, periodic all-at-once backups.

It's much cheaper to build your own than to use cloud storage esecially a secondary storage service like Dropbox. Even blackblaze B2 is $60/TB/year vs you can buy an 8TB HDD on Ebay for ~$200 and if the drive lasts 5 years that works out to be $5/TB/year and adding one or two redudent drives is still much cheaper.

How do you deal with the off-site problem? That's why most people use cloud services -- as an off-site backup that they don't need to manage themselves. Yeah, you could just put everything on an 8TB hard-drive from eBay (not sure that's a great idea for important data) and stick it on a shelf -- but the chance that drive will be alive and not have bit-rotted by the time you need the data is smaller than most people would be comfortable with.

Re: I Nearly Lost All of My Data

#192

Post by ashurov »

Synology allows syncing of your NAS to a similar Synoogy NAS on another location (Cloud station server). That's what I am doing. This doesn't prevent issues with data corruption, but prevents these kind of issues where you 'complete' NAS fails due to a possible surge.

Re: I Nearly Lost All of My Data

#193

Post by sreitshamer »

rsync wrote: post #135

Hard to believe we are 129 comments into this and there are zero mentions of 'borg' in the comment threads ... For those that don't know, borg is a backup utility[1] that has been called the "holy grail of backups"[2]. It takes your plaintext files and directories, chops them into gpg-encrypted chunks with encrypted, random filenames, and will upload (and maintain) them, with an efficient, changes-only update, to any…

Have you looked at Arq[1]? It deduplicates, encrypts, compresses, backs up to most major cloud providers and SFTP and Minio, is available for macOS and Windows. We've been improving it for 10 years. The data format is open and documented[2].

(Note: I work on Arq)

[1] https://www.arqbackup.com [2] https://www.arqbackup.com/arq_data_format.txt

Re: I Nearly Lost All of My Data

#194

Post by RealityVoid »

albertzeyer wrote: post #190

I recently was also thinking about how to organize my data & backups. I still have not really decided. Esp about the software. I collected lots of options here: https://github.com/albertz/wiki/blob/master/backup-software.... At the moment, I really like Perkeep ( https://perkeep.org/ ). But I'm not sure whether this is a solution for everything. On the hardware side, I also have not really decided. I want to build up…

Yes, I was also thinking about perkeep and giving it a spin. I was considering a RAID 1 array for redundancy.

Another thing I was considering was M-Disk[1], that can, supposedly, hold information for 1000 years. But I wonder what other people's experience with it is.

Alternatively, I understand normal Blu-Ray disks should be able to have data retention of about 40 years, and that sounds decent to me.

I was looking into Amazon glacier and/or google cold line, and the prices seem decent, but I do not like the fact that you have to pay monthly, even if a small sum, it's just one more thing to concern yourself with. I would like to prepay and know my stuff is up there for a couple of years.

Normal Dropbox/Google drive stuff is too expensive to store big amounts of data, so not worth it. Plus, you have a copy of it locally also, (at least in normal use cases)

Thinking about it, I think this would be a great idea for a startup. Cheap data storage for long term storage, with competitive prices, the ability to per-pay, user-side encryption and a simple UI that grandma can use to drop photos. It should be able to guarantee that the information you desire to will outlive you.

[1] https://en.wikipedia.org/wiki/M-DISC

Re: I Nearly Lost All of My Data

#195

Post by pimeys »

albertzeyer wrote: post #190

I recently was also thinking about how to organize my data & backups. I still have not really decided. Esp about the software. I collected lots of options here: https://github.com/albertz/wiki/blob/master/backup-software.... At the moment, I really like Perkeep ( https://perkeep.org/ ). But I'm not sure whether this is a solution for everything. On the hardware side, I also have not really decided. I want to build up…

I settled with a SuperMicro's passive Atom solution and a Fractal Design mini case, able to hold six disks. It is very quiet when you tune down the fans and keeps the disks just below 40°C which is optimal for me. We have the NAS in the same room we sleep and it doesn't bother us.

Re: I Nearly Lost All of My Data

#196

Post by Youden »

sreitshamer wrote: post #193
rsync wrote: post #135

Hard to believe we are 129 comments into this and there are zero mentions of 'borg' in the comment threads ... For those that don't know, borg is a backup utility[1] that has been called the "holy grail of backups"[2]. It takes your plaintext files and directories, chops them into gpg-encrypted chunks with encrypted, random filenames, and will upload (and maintain) them, with an efficient, changes-only update, to any…

Have you looked at Arq[1]? It deduplicates, encrypts, compresses, backs up to most major cloud providers and SFTP and Minio, is available for macOS and Windows. We've been improving it for 10 years. The data format is open and documented[2]. (Note: I work on Arq) [1] https://www.arqbackup.com [2] https://www.arqbackup.com/arq_data_format.txt

You've probably heard this before but there are a lot of people who'd be interested in a cross-platform (NAS/Linux/BSD/Raspberry Pi) version with a command-line or web interface.

As is it's really best for workstations rather than the NAS/servers a lot of people who care about their data use (not just basement-dwelling data hoarders but also professionals like photographers and videographers).

Re: I Nearly Lost All of My Data

#197

Post by nvr219 »

gtirloni wrote: post #24
derekp7 wrote: post #13

The best solution I found as an alternative to an off-site copy is to set up a USB drive and something like a Raspberry Pi in my car. When I pull into the garage, the house server senses it and syncs new data to it. This can be supplemented further by having it auto sync to a computer in your office whenever you pull into work. Add in some monitoring so you can get a phone alert if any of the synced copies are more t…

Does the RaspberryPi have enough CPU for disk encryption?

1. yes 2. you would want to encrypt before sending to rpi anyway

Re: I Nearly Lost All of My Data

#198

Post by doublepg23 »

petre wrote: post #150
Karunamon wrote: post #17

Earlier quoted context omitted.

The nice thing about Synology storage stuff is that it's a nice GUI (seriously, they're about the only company I can think of that's doing appliance management right) on top of standard and battle-tested open source tools. This was one of the reasons I was okay with paying their prices. Even if the device completely craps the bed, I'll be able to hook the drives containing absolutely normal LVM/btrfs volumes up to an…

Btrfs is battle tested? https://btrfs.wiki.kernel.org/index.php/Gotchas We use a box with 2x2TB mirror zpools. It runs Nas4free off an 128GB SSD. Initially we wanted to use Freenas but ir needs 16GB of RAM, while Nas4free works fine with 8. It also does SMART monitoring sending emails when something is up.

FreeNAS works fine with 8GB RAM, especially with such little data

Re: I Nearly Lost All of My Data

#199

Post by m3nu »

blattimwind wrote: post #187
rsync wrote: post #135

Hard to believe we are 129 comments into this and there are zero mentions of 'borg' in the comment threads ... For those that don't know, borg is a backup utility[1] that has been called the "holy grail of backups"[2]. It takes your plaintext files and directories, chops them into gpg-encrypted chunks with encrypted, random filenames, and will upload (and maintain) them, with an efficient, changes-only update, to any…

The crypto in Borg is bad. Even has its own page in the docs about how bad it really is. The tl;dr is "there is one AES key per repository and we use this with AES-CTR and trust the remote server to not fiddle with the counter". https://borgbackup.readthedocs.io/en/stable/internals/securi... Last time I checked there was still no work being done on improving performance. Multithreading has been on the agenda for year…

Like many open source projects, Borg is low on developers. The two points you mention are both known and on the agenda for this year. Since you seem to have experience with crypto, why not add a PR? It's all in Python and fairly readable.

Re: I Nearly Lost All of My Data

#200

Post by Kye »

m3nu wrote: post #152
rsync wrote: post #135

Hard to believe we are 129 comments into this and there are zero mentions of 'borg' in the comment threads ... For those that don't know, borg is a backup utility[1] that has been called the "holy grail of backups"[2]. It takes your plaintext files and directories, chops them into gpg-encrypted chunks with encrypted, random filenames, and will upload (and maintain) them, with an efficient, changes-only update, to any…

Let's also mention Vorta, an open source GUI for Borg to make it as easy to use as commercial backup tools. We just started translating into different languages. All input and PRs are welcome. https://github.com/borgbase/vorta/ (I'm the original author, but Thomas, the current Borg maintainer is also very active.)

There seem to be a lot of open source projects named after things in Star Trek.
Post reply on HN