I lost my data trying to back it up • Hacker News Board

Re: I lost my data trying to back it up

#101

Post by rikkus » Thu, Jan 31, 2019, 9:05 PM UTC

AdmiralAsshat wrote: post #51

rikkus wrote: post #43

Earlier quoted context omitted.

I use Code42 CrashPlan[0]. This has clients for MacOS, Linux and Windows and it's been flawless since I started using it years ago for personal and business use. Going by the names and the descriptions I've read, the overloaded term 'backup' only applies to Carbon Copy Cloner and Clonezilla in the sense that they make a copy of your storage, so don't protect against corruption (such as ransomware, or you accidentally…

CrashPlan for personal use no longer exists. Only the business license remains.

Yes, that's what I'm using. You don't have to be a business to use it.

Re: I lost my data trying to back it up

#102

Post by guitarbill » Thu, Jan 31, 2019, 9:06 PM UTC

HenryBemis wrote: post #89

guitarbill wrote: post #67

Earlier quoted context omitted.

For macOS, Time Machine can do automatic backups to a mounted NAS volume over WiFi. So my Macbook backs up to a ZFS 3-way mirror. Important files are also stored in DropBox. That's how I get relatively hassle free 3-2-1 backups. For Windows, a similar strategy is possible, but I consider my Windows box disposable (only gaming), so don't have much recent experience.

Just a risk I see here, you have everything IN your home.. laptop, backup. Unless your NAS is somewhere (not your home's internal wifi network but a WLAN). Consider a Carbonite (or similar) that has infinite backup space to make sure you have the 'offsite' covered.

Dropbox solves that to a certain degree, too, and provides cross OS sync for the files I care about most.

Honestly though, low effort is very important for me. After all, any solution is simply reducing the odds, not completely eliminating them.

Re: I lost my data trying to back it up

#103

Post by et-al » Thu, Jan 31, 2019, 9:07 PM UTC

simcop2387 wrote: post #97

coldtea wrote: post #93

> When it comes to backing up the full OS, I don’t believe in online backup solutions. And how did that work out for you? If you did believe in online backup solutions, in addition to whatever else, you'd have your data now. (I understand worrying about quick recovery, booting, etc. But, as an additional method besides hard local copies, it makes sense, as having an extra off site backup of the data is important. Bet…

This is how I treat it too. Always have a local backup, but an online or offsite one is also important. The mantra I've often heard repeated is "Two is one, and one is none".

Two is one; one is none is more of a military slogan. Backups should follow the 3-2-1 mantra, fullstop:

3 copies, on 2 different media, and 1 offsite.

Re: I lost my data trying to back it up

#104

Post by tubbs » Thu, Jan 31, 2019, 9:09 PM UTC

h4myio wrote: post #14

mobilemidget wrote: post #4

(many others including myself, have been there, different way though for me) The bright side is, you'll never make this mistake again. The downside is, for me at least, I end up with triple+ backups of everything..

I can defiantly see myself ending up with triple+ backups of everything...

Not to be a grammar Nazi, but I believe the word you're looking for is definitely. I wouldn't have normally picked it out, but I did see you use it like that in the article as well.

It was a good read! Thanks for taking your time to write it up.

Re: I lost my data trying to back it up

#105

Post by tracker1 » Thu, Jan 31, 2019, 9:14 PM UTC

arghwhat wrote: post #91

tracker1 wrote: post #54

Raid is not backup... Your data isn't backed up unless there are at least 3 full copies in 2 physical locations. I have to admit I'm slightly paranoid about it... 20-something years ago I was working in tech support for iomega (mostly os/2 and jazz calls). Nothing taught me the importance of true backups more than that job. I do leverage a few things though... for the most part, I keep stuff on my nas. Projects are i…

I agree with your idea, but your definition is overkill. A backup is a copy of data on hardware separate from the live system. A good backup is a backup in in a remote location. A backup strategy is to maintain a good backup. A good backup strategy is to have routine restore tests, to make sure that the backups serve their purpose. A great backup strategy requires at least 2 good backups.

I understand... by 3 full copies, that can include the active in-use copy. I just don't consider something backed up short of having the data in a separate device and location.

Re: I lost my data trying to back it up

#106

Post by jrockway » Thu, Jan 31, 2019, 9:30 PM UTC

et-al wrote: post #103

simcop2387 wrote: post #97

Earlier quoted context omitted.

This is how I treat it too. Always have a local backup, but an online or offsite one is also important. The mantra I've often heard repeated is "Two is one, and one is none".

Two is one; one is none is more of a military slogan. Backups should follow the 3-2-1 mantra, fullstop: 3 copies, on 2 different media, and 1 offsite.

I generalize this to saying "have N+1 replicas, where N is the number of simultaneous failures you want to survive". I'm perfectly happy having my .emacs backed up in 1 place, GitHub. If all my computers and GitHub lose data at the exact same time, I will just make new key bindings. The same might not hold true for your family heirlooms or company's invoices or whatever. Just make the right decision for your application, and don't trust any one system too much.

I also feel like adding, replication is less likely to work correctly the more abstract it tries to be. Compare hardware RAID (pretends to the rest of the computer that there is one disk instead of two) to application-level replication (don't return success until the document has been written successfully to 3 nodes). That first one never works correctly, because there's so much that can go wrong and so little you can do to fix it. The on-disk format depends on some random firmware that is treated as disposable by the manufacturer of your motherboard; with no documentation or source code to be had. The second one pretty much always works because it's simple, and when it breaks, you can just examine the content of each node and figure out what went wrong, because you designed it and it's as simple as it can possibly be. (If you want to not store 3 copies of your data, you add your own Reed-Solomon coding to the data you write, and pick the ratio of data size to lost chunks to suit your needs. That's what RAID-5 is, just abstracted over an entire POSIX filesystem and hundreds of thousands of lines of code with no unit tests.)

I would never use RAID again, as I consider the complexity too high for the benefits. Treat filesystems and disks as disposable and transient; don't try to build a filesystem that's durable.

Re: I lost my data trying to back it up

#107

Post by nikanj » Thu, Jan 31, 2019, 9:36 PM UTC

zaarn wrote: post #18

I think the moral of the story is; always use software RAID. Hardware RAID (even fake ones like Intel) are time bombs, either you have fun configurations like this or the controller dies and no replacement exists. Unless you'd absolutely need a HW RAID, I'd go for a Software RAID that mounts and works natively in Linux. I can put the harddrives into a fully foreign computer and it would work.

If anything, the moral of the story seems to be ”RAID is way too complex and dangerous for the benefits it brings”.

For home desktops, that is. Server end is different.

Re: I lost my data trying to back it up

#108

Post by pnutjam » Thu, Jan 31, 2019, 9:47 PM UTC

pault wrote: post #90

pnutjam wrote: post #53

Earlier quoted context omitted.

If the article writer didn't have 2 copies of his data, he didn't have a backup. I keep a running linux server that all my desktops, phones, and other devices back up to, it's a sort of home NAS. I have a 2nd internal drive that gets mounted and all the important photes and less personal stuff gets rsynced to it. My main drive is XFS, but the backup drive is btrfs. I use btrfs to maintain snapshots and unmount the dr…

I had several copies of my digital archive going back to 1996, including all of the art and music I made as a kid, as well as the first programs I wrote. A few weeks ago I reformatted one of my backup drives so I could use it with my roku TV to watch The Crimes of Grindlewald. Shortly after formatting the disk I realized that I had been mistaken and it was actually the only copy. The worst part is, the movie wasn't e…

You might be able to recover some of your info with testdisk, a linux utility. Download the system rescue cd (www.system-rescue-cd.org) if you need a rescue disk.

Re: I lost my data trying to back it up

#109

Post by cellularmitosis » Thu, Jan 31, 2019, 9:47 PM UTC

I think the key mistake here was allowing the complexity of his higher-level filesystem arrangements (raid, etc) to leak into his lower-level backup routine.

Making a system backup like this should just be about imaging the raw disks, regardless of OS, filesystem, raid arrangement, etc. sda is sda is sda, period.

Keep it simple and you drastically reduce the likelihood of making a mistake like this. A copy of Knoppix and dd will do the job just fine. Paragon Software, Clonezilla, ntfs-3g, mdadm -- distractions.

Even better, if you keep the fundamental units of your system simple (a raw image of sda), you maximize tool compatibility / availability and your chances of recovering when things aren't going to plan.

Re: I lost my data trying to back it up

#110

Post by Izkata » Thu, Jan 31, 2019, 10:46 PM UTC

tomschwiha wrote: post #58

mnw21cam wrote: post #48

Earlier quoted context omitted.

I have a server with 300TB of data. Incremental is the only way that my backups are going to be able to run every night. But also, there are different styles of incremental. Some are dodgier than others.

I agree for this volume of data incremental is the best choice. My projects so far ranged from 1-100GB so a full daily backup is quite quick done and its more easy to recover compared to applying incremental backups individually. Well, always choose the right tool for the task.

> compared to applying incremental backups individually.

At least one style of incremental backups can be restored in a single copy - I use rsync to create hardlinks to the previous day if the file is unchanged (based on modified time). This results in around 400M of additional disk usage per day due to changes, for around 90G of data in my home directory.