Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloning a Laptop over NVMe TCP

copyninja.in

Post reply on HN
Print view View original

1–10 of 181 posts

Re: Cloning a Laptop over NVMe TCP

#2

Post by transpute »

Thanks AWS/Annapurna/Nitro/Lightbits for bringing NVMe-over-TCP to Linux.

https://www.techtarget.com/searchstorage/news/252459311/Ligh...

> The NVM Express consortium ratified NVMe/TCP as a binding transport layer in November 2018. The standard evolved from a code base originally submitted to NVM Express by Lightbits' engineering team.

https://www.lightbitslabs.com/blog/linux-distributions-nvme-...

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Re: Cloning a Laptop over NVMe TCP

#4

Post by hcfman »

hcfman wrote: post #3

Very nice read! I didn't know it existed. Love it.

I love your explanation of your disk install with cryptsetup etc. I do a manual install as well as I always install with both mirrored and encrypted disks. The combination of the two I didn't find as easy install options (I think not at all) on the Ubuntu's I installed. Good to see a lot of this low level OS talk here.

Re: Cloning a Laptop over NVMe TCP

#5

Post by mrb »

In the author's scenario, there are zero benefits in using NVMe/TCP, as he just ends up doing a serial block copy using dd(1) so he's not leveraging concurrent I/O. All the complex commands can be replaced by a simple netcat.

On the destination laptop:

  $ nc -l -p 1234 | dd of=/dev/nvme0nX bs=1M
On the source laptop:

  $ nc x.x.x.x 1234
The dd on the destination is just to buffer writes so they are faster/more efficient. Add a gzip/gunzip on the source/destination and the whole operation is a lot faster if your disk isn't full, ie. if you have many zero blocks. This is by far my favorite way to image a PC over the network. I have done this many times. Be sure to pass "--fast" to gzip as the compression is typically a bottleneck on GigE. Or better: replace gzip/gunzip with lz4/unlz4 as it's even faster. Last time I did this was to image a brand new Windows laptop with a 1TB NVMe. Took 20 min (IIRC?) over GigE and the resulting image was 20GB as the empty disk space compresses to practically nothing. I typically back up that lz4 image and years later when I donate the laptop I restore the image with unlz4 | dd. Super convenient.

That said I didn't know about that Linux kernel module nvme-tcp. We learn new things every day :) I see that its utility is more for mounting a filesystem over a remote NVMe, rather than accessing it raw with dd.

Edit: on Linux the maximum pipe buffer size is 64kB so the dd bs=X argument doesn't technically need to be larger than that. But bs=1M doesn't hurt (it buffers the 64kB reads until 1MB has been received) and it's future-proof if the pipe sizes is ever increased :) Some versions of netcat have options to control the input and output block size which would alleviate the need to use dd bs=X but on rescue discs the netcat binary is usually a version without these options.

Re: Cloning a Laptop over NVMe TCP

#6

Post by wjnc »

So this just bit for bit dumps a NVMe device to another location. That’s clear. So all encryption just is transferred and not touched. But doesn’t the next machine go into panic when you boot? There are probably many changes in the underlying machine? (Okay, now I read the other post. The author really knows the way. This is at least intermediate Linux.)

Re: Cloning a Laptop over NVMe TCP

#8

Post by roomey »

I recently had to set up a new laptop (xubuntu).

Previously I cloned but I this time I wanted to refresh some of the configs.

Using a usb-c cable to transfer at 10gb/s is so useful (as my only other option was WiFi).

When you plug the computers together they form an ad-hoc network and you can just rsync across. As far as I could tell the link was saturated so using anything else (other protocols) would be pointless. Well not pointless, it's really good to learn new stuff, maybe just not when you are cloning your laptop (joke)!

Re: Cloning a Laptop over NVMe TCP

#9

Post by Gabrys1 »

I usually set up an initial distro and then copy over my /home. Later, I just need to install the debs I'm missing, but this has the benefit of not installing stuff I don't need anymore.

That said, I didn't know you could export NVMe over TCP like that, so still a nice read!

Post reply on HN