A snapshot of your computer with dd, pv and gzip

Re: A snapshot of your computer with dd, pv and gzip - Part 1

#11

Post by WizzleKake » Fri, Jul 19, 2013, 4:37 PM UTC

This is an incredibly naive way of backing up your system.

I use rsync to back up certain folders. I'm OK with re-installing Linux if my hard drive crashes.

Re: A snapshot of your computer with dd, pv and gzip - Part 1

#12

Post by sandycheeks » Fri, Jul 19, 2013, 4:38 PM UTC

networked wrote: post #6

Consider using Clonezilla [1] instead. While dd will produce a sector-by-sector image of your medium (which can be terribly inefficient if you're using considerably less than 100% of its available space) Clonezilla is file system-aware for a number of common file systems and has built-in support for compression as well. It resembles the recently discontinued Norton Ghost this way but is fully FOSS. I normally run it…

I would be concerned about how dd handles a bad sector that has not been reallocated by SMART. Many drives have 1 or 2 of these. Clonezilla is a good option because it also includes GNU ddrescue (so does PartedMagic) which can cope with most file system damage like bad sectors.

Re: A snapshot of your computer with dd, pv and gzip - Part 1

#13

Post by bcoates » Fri, Jul 19, 2013, 4:40 PM UTC

If you want to do something less tricky and low level than dd just remount the drive in question read-only and use tar. (be root to keep permissions, --use-compress-program=pigz)

On the restore side, you can just use pv like cat, and don't have to provide a size param:

  pv my.img.gz | pigz -d | dd (or hopefully tar xf - instead)...

In an ideal world a tool like pv could just look at /proc/fd/fdinfo/ instead of shuffling all the data through itself but I haven't found a tool that clever yet.

Re: A snapshot of your computer with dd, pv and gzip - Part 1

#14

Post by rlpb » Fri, Jul 19, 2013, 4:53 PM UTC

Remember that this cannot be run on a live system, since you'll get a corrupt image. Thus it's more useful for infrequent maintenance tasks than as part of a regular regime.

Re: A snapshot of your computer with dd, pv and gzip - Part 1

#15

Post by tlongren » Fri, Jul 19, 2013, 4:59 PM UTC

Pigz looks nice. First I've ever heard of it.

I just use BackInTime though, like TimeMachine I guess. It's really, really useful.

http://backintime.le-web.org/screenshots/

Re: A snapshot of your computer with dd, pv and gzip - Part 1

#16

Post by WestCoastJustin » Fri, Jul 19, 2013, 5:10 PM UTC

res0nat0r wrote: post #8

networked wrote: post #6

Consider using Clonezilla [1] instead. While dd will produce a sector-by-sector image of your medium (which can be terribly inefficient if you're using considerably less than 100% of its available space) Clonezilla is file system-aware for a number of common file systems and has built-in support for compression as well. It resembles the recently discontinued Norton Ghost this way but is fully FOSS. I normally run it…

I'd also recommend SysRescueCD that has these types of tools ready to be used as a bootable USB/CDrom image. http://www.sysresccd.org/System-tools I've used the imaging tools on the boot cd a few times to clone/restore hosts a few times without any issues. Very handy.

I created a SystemRescueCd screencast a while ago if anyone is interested [1]. Although, it does not look like SysRescueCD includes Clonezilla, there are tools like rsync, dd, and scp for moving files around in a pinch.

[1] http://sysadmincasts.com/episodes/3-systemrescuecd

Re: A snapshot of your computer with dd, pv and gzip - Part 1

#17

Post by j_s » Fri, Jul 19, 2013, 5:15 PM UTC

I used dd booting from a USB stick to transfer a Windows install from hard drive to SSD this weekend - there are a number of steps that can be done beforehand to tighten up the size of the final output. These steps are also useful for virtual machines, but they tend to take quite a while...

(1) Defrag - built-in / maybe SysInternals contig http://technet.microsoft.com/en-us/sysinternals/bb897428

(2) Defrag the page file - SysInternals PageDefrag http://technet.microsoft.com/en-us/sysinternals/bb897426

(3) Zero unused space - SysInternals SDelete http://technet.microsoft.com/en-us/sysinternals/bb897443.asp...

(4) Shrink volume (if backup is filesystem-aware, and disk is one large volume) - http://technet.microsoft.com/en-us/magazine/gg309169.aspx

(5) After transfer, expand volume to fit target disk

Pretty much any utility automating these steps is acquired by the various virtualization vendors, or heads directly to the enterprise market (kind of like how inexpensive screencast creation software always disappears!).

Re: A snapshot of your computer with dd, pv and gzip - Part 1

#18

Post by rsync » Fri, Jul 19, 2013, 5:21 PM UTC

I don't always backup an entire system with 'dd', but when I do:

dd if=/dev/da0 | ssh user@rsync.net "dd of=backup.dd"

or maybe:

pg_dump -U postgres db | ssh user@rsync.net "dd of=db_dump"

Re: A snapshot of your computer with dd, pv and gzip - Part 1

#19

Post by relaxitup » Fri, Jul 19, 2013, 5:25 PM UTC

WizzleKake wrote: post #11

This is an incredibly naive way of backing up your system. I use rsync to back up certain folders. I'm OK with re-installing Linux if my hard drive crashes.

It's not naive at all. There are a plethora of situations/reasons where one may want or need a block level image backup.

Re: A snapshot of your computer with dd, pv and gzip - Part 1

#20

Post by lttlrck » Fri, Jul 19, 2013, 5:28 PM UTC

rsync wrote: post #18

I don't always backup an entire system with 'dd', but when I do: dd if=/dev/da0 | ssh user@rsync.net "dd of=backup.dd" or maybe: pg_dump -U postgres db | ssh user@rsync.net "dd of=db_dump"

"Stay thirsty, my friends."