I lost my data trying to back it up • Hacker News Board

Re: I lost my data trying to back it up

#61

Post by majewsky » Thu, Jan 31, 2019, 4:06 PM UTC

anilakar wrote: post #30

zaarn wrote: post #18

I think the moral of the story is; always use software RAID. Hardware RAID (even fake ones like Intel) are time bombs, either you have fun configurations like this or the controller dies and no replacement exists. Unless you'd absolutely need a HW RAID, I'd go for a Software RAID that mounts and works natively in Linux. I can put the harddrives into a fully foreign computer and it would work.

I had this happen eleven years ago, when an integrated hardware RAID controller in a Dell server decided to literally blow up. Oddly enough, despite all the black burn marks on the mainboard, rest of the server seemed to work just okay. Another identical (hardware-wise) cold spare was already in the rack right below the unlucky one, but it wouldn't accept the transplant license dongle because of service tag mismatch.…

Lesson learned: Backups don't count until you've actually tested your restore procedure.

Re: I lost my data trying to back it up

#62

Post by moviuro » Thu, Jan 31, 2019, 4:08 PM UTC

Redundancy

I don't have backups myself, but I have 5 copies of my data at 3 different locations + snapshots everywhere: desktop, laptop (linux, BTRFS), NAS at home (FreeBSD, ZFS); off-site dedicated server at OVH (FreeBSD, ZFS); at school (FreeBSD, ZFS/netapp, managed by school IT).

Syncthing [0] + whatever handles local snapshots [1,2] work wonders [3]

[0] https://syncthing.net

[1] https://github.com/zfsnap/zfsnap

[2] https://gitlab.com/moviuro/butter

[3] https://try.popho.be/securing-home.html

Re: I lost my data trying to back it up

#63

Post by evanweaver » Thu, Jan 31, 2019, 4:21 PM UTC

> By doing so, I effectively set off a time bomb. That single sector stayed around, lurking, waiting for the right moment.

Technically this is a booby trap, not a time bomb. ;-)

Re: I lost my data trying to back it up

#64

Post by evanweaver » Thu, Jan 31, 2019, 4:50 PM UTC

kkarakk wrote: post #3

why would you try this out with "live" data? why not incrementally connect drives and do it that way? i'm basically asking what the impetus was for "just going for it" which is probably going to be a reason that other people share...

This is a good point; he could have physically removed one pair of drives, at least, at which point he would have had an actual backup sitting on the shelf.

Re: I lost my data trying to back it up

#65

Post by saint_abroad » Thu, Jan 31, 2019, 5:04 PM UTC

It's commendable that the OP was looking to backup but has the reliability of RAID against disk failure contributed to this being a "black swan" event?

Do we reduce failure rates to "acceptable" at the expense of neglecting recovery?

Re: I lost my data trying to back it up

#66

Post by [deleted] » Thu, Jan 31, 2019, 5:23 PM UTC

shifto wrote: post #44

zaarn wrote: post #18

I think the moral of the story is; always use software RAID. Hardware RAID (even fake ones like Intel) are time bombs, either you have fun configurations like this or the controller dies and no replacement exists. Unless you'd absolutely need a HW RAID, I'd go for a Software RAID that mounts and works natively in Linux. I can put the harddrives into a fully foreign computer and it would work.

Depends on the use-case really. I use raid (if any) for performance reasons, so I prefer hardware raid with a nice controller and some lots of spindles where flash is not an option. I would never implement this with a software raid solution for reasons.

[deleted]

Re: I lost my data trying to back it up

#67

Post by guitarbill » Thu, Jan 31, 2019, 5:29 PM UTC

NKosmatos wrote: post #27

Sorry for your loss. Here are my two cents... For complete system backup on Windows use Acronis [0], on MacOS use Carbon Copy Cloner [1] and on Linux Clonezilla [2]. Also never forget the 3-2-1 rule of backups [3], 3 copies, 2 local copies on different mediums and 1 copy offsite (cloud, remote) :-) [0] https://www.acronis.com/en-us/personal/computer-backup/ [1] https://bombich.com/ [2] https://clonezilla.org/ [3] htt…

For macOS, Time Machine can do automatic backups to a mounted NAS volume over WiFi. So my Macbook backs up to a ZFS 3-way mirror. Important files are also stored in DropBox. That's how I get relatively hassle free 3-2-1 backups.

For Windows, a similar strategy is possible, but I consider my Windows box disposable (only gaming), so don't have much recent experience.

Re: I lost my data trying to back it up

#68

Post by guitarbill » Thu, Jan 31, 2019, 5:37 PM UTC

tomatocracy wrote: post #56

shifto wrote: post #44

Earlier quoted context omitted.

Depends on the use-case really. I use raid (if any) for performance reasons, so I prefer hardware raid with a nice controller and some lots of spindles where flash is not an option. I would never implement this with a software raid solution for reasons.

Likewise. If it’s not offline, verified and off-site, it’s not a backup. Raid is useful for availability as well as performance though. Also while we’re discussing raid, the price/gb of a mirrored set of larger disks vs a raid 5/6/10 of smaller disks (for the same total capacity) is always worth checking - usually comes out in favour of mirrored pair in my experience which also has better performance (especially when…

seconded - storage is cheap, and with ZFS, mirrors can be grown, wheras arrays can't. i recently grew a mirror from 3 TB to 8 TB. strongly recommend using a three-way mirror though. don't be cheap.

Re: I lost my data trying to back it up

#69

Post by eslaught » Thu, Jan 31, 2019, 5:41 PM UTC

newscracker wrote: post #49

olavgg wrote: post #10

Getting backup right is extremely important, that is why you need proper tools. For the last 10 years I have been using ZFS snapshots and replication. I keep 180 days with snapshots on the remote backup and 30 days for the local. I use it to take backup of virtual machines, different databases including "running", sql servers. And of course important files, documents and photos. I've also configured a daily mail with…

What exactly do you do to verify that whatever you backed up was indeed backed up bit-for-bit and that the backup medium didn't just report success even though it didn't get some data stored correctly? Do you verify every backup using some tool? If yes, how long does that take? The reason I ask is because so many sites talk about the benefits of backing up regularly, using backup rotation and retention policies, etc.…

It's somewhat basic right now, but I developed a tool that you can use as an additional level of verification that your backups are working.

https://github.com/elliottslaughter/integrity-checker

Re: I lost my data trying to back it up

#70

Post by guitarbill » Thu, Jan 31, 2019, 5:45 PM UTC

21 wrote: post #47

No mention of offsite backup. Article could have stopped there. More subtly, he is the epitome of what I call "cowboy programmer/sysadmin", who doesn't really know what he's doing, just always improvising the next step based on minimal research. Which is fine and productive, until you are dealing with critical data/systems. I've witnessed so many times during my career this kind of people royally fuck things up with…

Bit harsh dude. He admitted he was wrong and even wrote a post about it. That's very honest, and something we should commend.