Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloning a Laptop over NVMe TCP

copyninja.in

Post reply on HN
Print view View original

91–100 of 181 posts

Re: Cloning a Laptop over NVMe TCP

#91

Post by doubled112 »

mrb wrote: post #64
bayindirh wrote: post #59

Earlier quoted context omitted.

> Your example proves me right. Your drive should be capable of 1000 MB/s but O_DIRECT reduces performance to 400 MB/s. I noted that the bus I connected the device has 500MBps bandwidth theoretical, no? To cite myself: > Target is a Samsung T7 Shield 2TB, with 1050MB/sec sustained write speed. Bus is USB 3.0 with 500MBps top speed (so I can go %50 of drive speeds). Result is 404MBps, which is fair for the bus.

Yes USB3.0 is 500 MB/s but are you sure your bus is 3.0? It would imply your machine is 10+ years old. Most likely it's 3.1 or newer which is 1000 MB/s. And again, benchmarking /dev/zero is invalid anyway as I explained (transparent compression)

TIL they have been sneaking versions of USB in while I haven't been paying attention. Even on hardware I own. Thanks for that.

Re: Cloning a Laptop over NVMe TCP

#92

Post by coretx »

bayindirh wrote: post #28
coretx wrote: post #27

Doing this without systemd or directly from the netboot environment would be interesting.

The user didn't do it from systemd actually. Instead they booted GRML, which is not very different from a netboot environment, and hand-exported the device themselves.

GRML is a debian live image, much different from a netboot environment. Look at https://ipxe.org/, boot firmware doing fiber over ethernet, iSCSI, HTTP; booting from virtually everything but NVMe over TCP.

Re: Cloning a Laptop over NVMe TCP

#93

Post by bayindirh »

coretx wrote: post #92
bayindirh wrote: post #28

Earlier quoted context omitted.

The user didn't do it from systemd actually. Instead they booted GRML, which is not very different from a netboot environment, and hand-exported the device themselves.

GRML is a debian live image, much different from a netboot environment. Look at https://ipxe.org/ , boot firmware doing fiber over ethernet, iSCSI, HTTP; booting from virtually everything but NVMe over TCP.

I use GRML almost daily, sometimes booting it over network and loading it to RAM. We don't use iPXE specificially, but use "vanilla" PXE regularly, too, generally to install systems with Kickstart or xCAT, depending on the cluster we're working on.

I'll be using GRML again tonight to rescue some laptop drives which were retrieved from a fire, to see what can be salvaged, for example. First in forensic mode to see what's possible, then to use dd_rescue on a N95 box.

Re: Cloning a Laptop over NVMe TCP

#94

Post by streb-lo »

tarruda wrote: post #66
M95D wrote: post #47

I don't understand why he didn't pipe btrfs through network. Do a btrfs snapshot first, then btrfs send => nc => network => nc => brtfs receive. That way only blocks in use are sent.

That's the first thing I thought when I saw he used btrfs. I use btrfs send/receive all the time via SSH and it works great. He could easily have setup SSH server in GRML live session. There's one caveat though: With btrfs it is not possible to send snapshots recursively, so if he had lots of recursive snapshots (which can happen in Docker/LXD/Incus), it is relatively hard to mirror the same structure in a new disk.…

Yea, there needs to be a `snapshot -r` option or something. I like using subvolumes to manage what gets a snapshot but sometimes you want the full disk.

Re: Cloning a Laptop over NVMe TCP

#95

Post by klabb3 »

dmos62 wrote: post #23

I recently had to copy around 200gb of files over wifi. I used rsync to make sure a connection failure doesn't mean I have to start over and so that nothing is lost, but it took at least 6 hours. I wonder what could I have done better. Btw, what kinds of guarantees do you get with the dd method? Do you have to compare md5s of the resulting block level devices after?

> 200gb of files over wifi […] took at least 6 hours

> I wonder what could I have done better.

Used an Ethernet cable? That’s not an impressive throughput amount over local. WiFi has like a million more sources of perf bottlenecks. Btw, just using a cable on ONE of the device => router ~> device can help a lot.

Re: Cloning a Laptop over NVMe TCP

#96

Post by verticalscaler »

throwawaaarrgh wrote: post #85
rwmj wrote: post #37

A lot of hassle compared to: nbdkit file /dev/nvme0n1 nbdcopy nbd://otherlaptop localfile

This is actually much better because nbdcopy can handle sparse files, you can set the number of connections and threads to number of cores, you can force a flush before exit, and enable a progress bar. For unencrypted drives it also supports TLS.

If you really want a progress bar chuck in a 'pv' somewhere into the command posted at the top of the thread.

Re: Cloning a Laptop over NVMe TCP

#97

Post by ufocia »

mrb wrote: post #52
bayindirh wrote: post #45

Earlier quoted context omitted.

> the author's solution exposes his NVMe to unauthenticated remote write access by any number of clients(!) I won't be bothered in a home network. > Clonezilla are vastly more moving parts ...and one of these moving parts is image integrity and write integrity verification, allowing byte-by-byte integrity during imaging and after write. > I strongly recommend against oflag=direct as in this... [snipped for brevity] U…

Your example proves me right. Your drive should be capable of 1000 MB/s but O_DIRECT reduces performance to 400 MB/s. This matters in the specific use case of "netcat | gunzip | dd" as the compressed data rate on GigE will indeed be around 120 MB/s but when gunzip is decompressing unused parts of the filesystem (which compress very well), it will attempt to write 1+ GB/s or more to the pipe to dd and it would not be…

I wonder how using tee to compute the hash in parallel would affect the overall performance.

Re: Cloning a Laptop over NVMe TCP

#98

Post by fransje26 »

mrb wrote: post #5

In the author's scenario, there are zero benefits in using NVMe/TCP, as he just ends up doing a serial block copy using dd(1) so he's not leveraging concurrent I/O. All the complex commands can be replaced by a simple netcat. On the destination laptop: $ nc -l -p 1234 | dd of=/dev/nvme0nX bs=1M On the source laptop: $ nc x.x.x.x 1234 The dd on the destination is just to buffer writes so they are faster/more efficient…

I am not often speechless, but this hit the spot. Well done Sir!

Where does one learn this black art?

Re: Cloning a Laptop over NVMe TCP

#99

Post by fransje26 »

roomey wrote: post #8

I recently had to set up a new laptop (xubuntu). Previously I cloned but I this time I wanted to refresh some of the configs. Using a usb-c cable to transfer at 10gb/s is so useful (as my only other option was WiFi). When you plug the computers together they form an ad-hoc network and you can just rsync across. As far as I could tell the link was saturated so using anything else (other protocols) would be pointless.…

Were you transferring the entire filesystem, after booting from a live disk, or were you transferring files over after having set up a base system?

Re: Cloning a Laptop over NVMe TCP

#100

Post by raudette »

matrss wrote: post #72
netsharc wrote: post #19

Earlier quoted context omitted.

Yeah, I've done several such transplants of disks onto different hardware.. maybe even from an AMD to an Intel system once (different mainboard chipsets), and of Windows 7. As you say, it will say new hardware detected and a lot of the times will get the drivers for them from Windows Update. There are also 3rd party tools to remove no-longer existing devices from the registry, to clean the system up a bit.

Windows will, however, no longer acknowledge your license key. At least that is what happened when I swapped a Windows installation from an Intel system into an AMD system.

With Windows 10, I have unplugged the primary storage from an old laptop, and plugged it into a new one, without issue.

The system booted, no license issues.

Post reply on HN