Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I lost my data trying to back it up

hamy.io

Post reply on HN
Print view View original

31–40 of 126 posts

Re: I lost my data trying to back it up

#31

Post by anilakar »

ghaff wrote: post #19
zaarn wrote: post #18

I think the moral of the story is; always use software RAID. Hardware RAID (even fake ones like Intel) are time bombs, either you have fun configurations like this or the controller dies and no replacement exists. Unless you'd absolutely need a HW RAID, I'd go for a Software RAID that mounts and works natively in Linux. I can put the harddrives into a fully foreign computer and it would work.

The other moral is that RAID is not a backup. I’d also argue that for most individuals who don’t really need to worry about an hour or two of downtime, it’s a lot more useful to have good (multiple) current backups than potentially complex RAID configurations.

RAID is the zeroth-tier backup. At least it will prevent most of the horror stories where people accidentally destroy their backups when trying to restore them.

Re: I lost my data trying to back it up

#32

Post by olavgg »

fyfy18 wrote: post #22
olavgg wrote: post #10

Getting backup right is extremely important, that is why you need proper tools. For the last 10 years I have been using ZFS snapshots and replication. I keep 180 days with snapshots on the remote backup and 30 days for the local. I use it to take backup of virtual machines, different databases including "running", sql servers. And of course important files, documents and photos. I've also configured a daily mail with…

Do you use ZFS on the desktop too, or something different? I've used btrfs raid on a server before, and it worked apart one incident which took it two weeks (for 4TB of data) to recover from, so I'm probably going to move to ZFS on the next upgrade. On the desktop there seem to be many more tools for btrfs though, like timeshift [0] - effectively Time Machine for Linux. [0] https://github.com/teejee2008/timeshift

I use NFS, which works really great and is faster than what most people believe. With the exception of no local page cache which impact latency for some applications. iSCSI is another great alternative, and with iSCSI you will also benefit from your local page cache. Though only one machine can use one iSCSI volume at the time. NFS can be shared between different machines.

In my setup there is no special network tuning either, it is completely basic MTU is 1500, NO VLAN's, both iSCSI and regular network traffic works great without impacting for each other. Performance is excellent with the cheap Mellanox ConnextX-2 ethernet cards you find on Ebay for 20 USD. There are some really excellent, cheap, silent and with 10G SFP+ ports network switches available too for just 100USD

Re: I lost my data trying to back it up

#33

Post by chatmasta »

olavgg wrote: post #10

Getting backup right is extremely important, that is why you need proper tools. For the last 10 years I have been using ZFS snapshots and replication. I keep 180 days with snapshots on the remote backup and 30 days for the local. I use it to take backup of virtual machines, different databases including "running", sql servers. And of course important files, documents and photos. I've also configured a daily mail with…

Is it safe to backup a database by just straight copying its files? Wouldn't it be better to use a specific backup tool, e.g. pg_dump for Postgres or mongodump for Mongo?

(Hopefully you've tested recovery procedures, because as the saying goes, "your backups are only as good as your last restore")

Re: I lost my data trying to back it up

#35

Post by rikkus »

olavgg wrote: post #10

Getting backup right is extremely important, that is why you need proper tools. For the last 10 years I have been using ZFS snapshots and replication. I keep 180 days with snapshots on the remote backup and 30 days for the local. I use it to take backup of virtual machines, different databases including "running", sql servers. And of course important files, documents and photos. I've also configured a daily mail with…

If you check that you can restore from backup regularly, then whatever the system is underneath that matters much less.

Even an email saying 'things went well' doesn't tell you that you can restore your files.

Re: I lost my data trying to back it up

#36

Post by sokoloff »

cm2187 wrote: post #5

That sounds like a very complicated way to do a full OS backup. Dead easy, 10 min, can't fail alternative for home projects: use a VM instead, switch off the VM, copy the disk file, switch the VM back on. Use NVMe SSDs and depending on the size, you can probably do an automated daily backup with minimal downtime.

Anyone have a solution that can snapshot a running VM that has a PCI or PCIe card passed through to the guest OS? I use snapshots on ESXi, but they don't work for guests that have a card passed-through.

Re: I lost my data trying to back it up

#37

Post by pmlnr »

olavgg wrote: post #10

Getting backup right is extremely important, that is why you need proper tools. For the last 10 years I have been using ZFS snapshots and replication. I keep 180 days with snapshots on the remote backup and 30 days for the local. I use it to take backup of virtual machines, different databases including "running", sql servers. And of course important files, documents and photos. I've also configured a daily mail with…

ZFS is great, but it has it's own quirks sometimes. I had an extremely hard time getting rid of a broken zpool a while ago.

Re: I lost my data trying to back it up

#38

Post by pmlnr »

chatmasta wrote: post #33
olavgg wrote: post #10

Getting backup right is extremely important, that is why you need proper tools. For the last 10 years I have been using ZFS snapshots and replication. I keep 180 days with snapshots on the remote backup and 30 days for the local. I use it to take backup of virtual machines, different databases including "running", sql servers. And of course important files, documents and photos. I've also configured a daily mail with…

Is it safe to backup a database by just straight copying its files? Wouldn't it be better to use a specific backup tool, e.g. pg_dump for Postgres or mongodump for Mongo? (Hopefully you've tested recovery procedures, because as the saying goes, "your backups are only as good as your last restore")

ZFS is copy-on-write, so in most cases, it is safe.

Re: I lost my data trying to back it up

#39

Post by lmz »

chatmasta wrote: post #33
olavgg wrote: post #10

Getting backup right is extremely important, that is why you need proper tools. For the last 10 years I have been using ZFS snapshots and replication. I keep 180 days with snapshots on the remote backup and 30 days for the local. I use it to take backup of virtual machines, different databases including "running", sql servers. And of course important files, documents and photos. I've also configured a daily mail with…

Is it safe to backup a database by just straight copying its files? Wouldn't it be better to use a specific backup tool, e.g. pg_dump for Postgres or mongodump for Mongo? (Hopefully you've tested recovery procedures, because as the saying goes, "your backups are only as good as your last restore")

If you use a consistent snapshot of the volume it should be ok. The database would perform crash recovery on startup but data should still be consistent.

Re: I lost my data trying to back it up

#40

Post by madshiva »

This is the wrong way of doing backup.

Raid for backup is not so good, specially if you have only 2 drive...

Using software (all provided is too a not good example.)

Just copy to another hare drive (network, other machine) and mirroir rsync that damn folder, the cost is low and easy to setup and recover.

Post reply on HN