I lost my data trying to back it up • Hacker News Board

Re: I lost my data trying to back it up

#41

Post by chousuke » Thu, Jan 31, 2019, 1:26 PM UTC

chatmasta wrote: post #33

olavgg wrote: post #10

Getting backup right is extremely important, that is why you need proper tools. For the last 10 years I have been using ZFS snapshots and replication. I keep 180 days with snapshots on the remote backup and 30 days for the local. I use it to take backup of virtual machines, different databases including "running", sql servers. And of course important files, documents and photos. I've also configured a daily mail with…

Is it safe to backup a database by just straight copying its files? Wouldn't it be better to use a specific backup tool, e.g. pg_dump for Postgres or mongodump for Mongo? (Hopefully you've tested recovery procedures, because as the saying goes, "your backups are only as good as your last restore")

With Postgres, it's perfectly safe and supported if you just do it according to the documentation using pg_start/stop_backup and also archive your WALs properly. Most databases have something similar.

Re: I lost my data trying to back it up

#42

Post by lettergram » Thu, Jan 31, 2019, 1:36 PM UTC

chousuke wrote: post #41

chatmasta wrote: post #33

Earlier quoted context omitted.

Is it safe to backup a database by just straight copying its files? Wouldn't it be better to use a specific backup tool, e.g. pg_dump for Postgres or mongodump for Mongo? (Hopefully you've tested recovery procedures, because as the saying goes, "your backups are only as good as your last restore")

With Postgres, it's perfectly safe and supported if you just do it according to the documentation using pg_start/stop_backup and also archive your WALs properly. Most databases have something similar.

Note, you do need to track the version of Postgres. So a backup from a PG 8 may need to be reloaded into PG 8 before upgrading to PG 9, 10, 11, etc

Re: I lost my data trying to back it up

#43

Post by rikkus » Thu, Jan 31, 2019, 1:41 PM UTC

NKosmatos wrote: post #27

Sorry for your loss. Here are my two cents... For complete system backup on Windows use Acronis [0], on MacOS use Carbon Copy Cloner [1] and on Linux Clonezilla [2]. Also never forget the 3-2-1 rule of backups [3], 3 copies, 2 local copies on different mediums and 1 copy offsite (cloud, remote) :-) [0] https://www.acronis.com/en-us/personal/computer-backup/ [1] https://bombich.com/ [2] https://clonezilla.org/ [3] htt…

I use Code42 CrashPlan[0]. This has clients for MacOS, Linux and Windows and it's been flawless since I started using it years ago for personal and business use.

Going by the names and the descriptions I've read, the overloaded term 'backup' only applies to Carbon Copy Cloner and Clonezilla in the sense that they make a copy of your storage, so don't protect against corruption (such as ransomware, or you accidentally corrupting files, or deleting files...) Please let me know if I'm wrong on this. Of course you can do real backups with a clone-only tool if you rotate the backup media yourself, but I prefer this aspect to be handled automatically.

[0] https://www.crashplan.com

Re: I lost my data trying to back it up

#44

Post by shifto » Thu, Jan 31, 2019, 1:41 PM UTC

zaarn wrote: post #18

I think the moral of the story is; always use software RAID. Hardware RAID (even fake ones like Intel) are time bombs, either you have fun configurations like this or the controller dies and no replacement exists. Unless you'd absolutely need a HW RAID, I'd go for a Software RAID that mounts and works natively in Linux. I can put the harddrives into a fully foreign computer and it would work.

Depends on the use-case really. I use raid (if any) for performance reasons, so I prefer hardware raid with a nice controller and some lots of spindles where flash is not an option. I would never implement this with a software raid solution for reasons.

Re: I lost my data trying to back it up

#45

Post by mbell » Thu, Jan 31, 2019, 1:56 PM UTC

chatmasta wrote: post #33

olavgg wrote: post #10

Getting backup right is extremely important, that is why you need proper tools. For the last 10 years I have been using ZFS snapshots and replication. I keep 180 days with snapshots on the remote backup and 30 days for the local. I use it to take backup of virtual machines, different databases including "running", sql servers. And of course important files, documents and photos. I've also configured a daily mail with…

Is it safe to backup a database by just straight copying its files? Wouldn't it be better to use a specific backup tool, e.g. pg_dump for Postgres or mongodump for Mongo? (Hopefully you've tested recovery procedures, because as the saying goes, "your backups are only as good as your last restore")

Depends on the database / disk setup. For Mongo you should probably do an fsync/lock prior to taking the snapshot then release it, this will ensure the data on disk is in a consistent state. I don't think this is technically required with a single disk setup (at least with LVM, I've not used ZFS with mongo) but it's probably a good idea anyway and only causes a few ms hiccup.

Re: I lost my data trying to back it up

#46

Post by jondubois » Thu, Jan 31, 2019, 2:06 PM UTC

The last time I lost all most of my data was when I tried upgrading Ubuntu to the next major version using the upgrade manager in Ubuntu. I thought that since the UI was so friendly looking, it must be the safest route. WRONG. It was the second time that I trusted Ubuntu's UI for doing something critical. Never again. Now I always back up all the files I need onto an external drive and then I install the OS from scratch. Only trust the command line.

Re: I lost my data trying to back it up

#47

Post by 21 » Thu, Jan 31, 2019, 2:09 PM UTC

No mention of offsite backup. Article could have stopped there.

More subtly, he is the epitome of what I call "cowboy programmer/sysadmin", who doesn't really know what he's doing, just always improvising the next step based on minimal research. Which is fine and productive, until you are dealing with critical data/systems.

I've witnessed so many times during my career this kind of people royally fuck things up with this "I'm a terminal God, of course I'm doing it live on production" attitude.

Everybody I know who used hardware RAID for their personal setup ended up losing their data EVERY SINGLE TIME. Hardware RAID has zero tolerance for human error, it's like the Suicide Linux distribution.

Re: I lost my data trying to back it up

#48

Post by mnw21cam » Thu, Jan 31, 2019, 2:36 PM UTC

tomschwiha wrote: post #21

jacobush wrote: post #6

Earlier quoted context omitted.

Incrementally?

I assume by incrementally is meant to have a base backup and store the difference in between. Personally I prefer full backups - they feel more complete.

I have a server with 300TB of data. Incremental is the only way that my backups are going to be able to run every night.

But also, there are different styles of incremental. Some are dodgier than others.

Re: I lost my data trying to back it up

#49

Post by newscracker » Thu, Jan 31, 2019, 2:52 PM UTC

olavgg wrote: post #10

Getting backup right is extremely important, that is why you need proper tools. For the last 10 years I have been using ZFS snapshots and replication. I keep 180 days with snapshots on the remote backup and 30 days for the local. I use it to take backup of virtual machines, different databases including "running", sql servers. And of course important files, documents and photos. I've also configured a daily mail with…

What exactly do you do to verify that whatever you backed up was indeed backed up bit-for-bit and that the backup medium didn't just report success even though it didn't get some data stored correctly? Do you verify every backup using some tool? If yes, how long does that take? The reason I ask is because so many sites talk about the benefits of backing up regularly, using backup rotation and retention policies, etc., but they usually don't talk about verification of the backups themselves. Even many popular backup tools don't seem to handle this part (sometimes due to complexity introduced by buffer caches).

Re: I lost my data trying to back it up

#50

Post by giomasce » Thu, Jan 31, 2019, 2:56 PM UTC

To me the takeaway is that backup system should be designed with the storage system, when the storage system is still empty of valuable data. If you change the backup system, you should treat that as a change of the whole storage system, so first you would run the former backup, archive that securely, set up the new system, check it and only then release the former backup as not critical anymore.