Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloning a Laptop over NVMe TCP

copyninja.in

Post reply on HN
Print view View original

141–150 of 181 posts

Re: Cloning a Laptop over NVMe TCP

#141

Post by HankB99 »

mrb wrote: post #5

In the author's scenario, there are zero benefits in using NVMe/TCP, as he just ends up doing a serial block copy using dd(1) so he's not leveraging concurrent I/O. All the complex commands can be replaced by a simple netcat. On the destination laptop: $ nc -l -p 1234 | dd of=/dev/nvme0nX bs=1M On the source laptop: $ nc x.x.x.x 1234 The dd on the destination is just to buffer writes so they are faster/more efficient…

I came here to suggest similar. I usually go with

    dd if=/dev/device | mbuffer to Ethernet to mbuffer dd of=/dev/device
(with some switches to select better block size and tell mbuffer to send/receive from a TCP socket)

If it's on a system with a fast enough processor I can save considerable time by compressing the stream over the network connection. This is particularly true when sending a relatively fresh installation where lots of the space on the source is zeroes.

Re: Cloning a Laptop over NVMe TCP

#142

Post by jeroenhd »

Moneysac wrote: post #13
jeroenhd wrote: post #11

I'm sure there are benefits to this approach, but I've transferred laptops before by launching an installer on both and then combining dd and nc on both ends. If I recall correctly, I also added gzip to the mix to make transferring large null sections faster. With the author not having access to an ethernet port on the new laptop, I think my hacky approach might've even been faster because of the slight boost compres…

could you explain how you do that exactly?

Basically https://news.ycombinator.com/item?id=39676881, but also adding a pipe through gzip/gunzip in the middle for data compression.

I think I did something like `nc -l -p 1234 | gunzip | dd status=progress of=/dev/nvme0n1` on the receiving end and `dd if=/dev/nvme0n1 bs=40M status=progress | gzip | nc 10.1.2.3:1234` on the sending end, after plugging an ethernet cable into both devices. In theory I could've probably also used the WiFi cards to set up a point to point network to speed up the transmission, but I couldn't be bothered with looking up how to make nc use mptcp like that.

Re: Cloning a Laptop over NVMe TCP

#143

Post by downrightmike »

dmos62 wrote: post #23

I recently had to copy around 200gb of files over wifi. I used rsync to make sure a connection failure doesn't mean I have to start over and so that nothing is lost, but it took at least 6 hours. I wonder what could I have done better. Btw, what kinds of guarantees do you get with the dd method? Do you have to compare md5s of the resulting block level devices after?

Wifi shares the medium (air) with all other radios. It has a random time it waits after it stops if it sees a collision.

"Carrier-sense multiple access with collision avoidance (CSMA/CA) in computer networking, is a network multiple access method in which carrier sensing is used, but nodes attempt to avoid collisions by beginning transmission only after the channel is sensed to be "idle".[1][2] When they do transmit, nodes transmit their packet data in its entirety.

It is particularly important for wireless networks, where the alternative with collision detection CSMA/CD, is not possible due to wireless transmitters desensing (turning off) their receivers during packet transmission.

CSMA/CA is unreliable due to the hidden node problem.[3][4]

CSMA/CA is a protocol that operates in the data link layer."

https://en.wikipedia.org/wiki/Carrier-sense_multiple_access_...

Re: Cloning a Laptop over NVMe TCP

#144

Post by wtallis »

PeterisP wrote: post #135
JoeAltmaier wrote: post #107

Earlier quoted context omitted.

Can't find corroboration for the assertion 'SSDs are an awful lot better at sequential writes than random writes'. Doesn't make sense at first glance. There's no head to move, as in an old-style hard drive. What else could make random write take longer on an SSD?

The key aspect is that such memory generally works on a "block" level so making any smaller-than-block write on a SSD requires reading a whole block (which can be quite large), erasing that whole block and then writing back the whole modified block; as you physically can't toggle a bit without erasing the whole block first. So if large sequential writes mean that you only write full whole blocks, that can be done muc…

In practice, flash based SSDs basically never do a full read-modify-write cycle to do an in-place update of an entire erase block. They just write the new data elsewhere and keep track of the fragmentation (consequently, sequential reads of data that wasn't written sequentially may not be as fast as sequential reads of data that was written sequentially).

RMW cycles (though not in-place) are common for writes smaller than a NAND page (eg. 16kB) and basically unavoidable for writes smaller than the FTL's 4kB granularity

Re: Cloning a Laptop over NVMe TCP

#145

Post by timetraveller26 »

I was in a situation like this recently, and just want to say that if opening the pc/laptop and connecting it to the new machine is possible you should go that route, it's a little hassle compared to wait to a transfer over the network, waiting times can be deceptive.

This reminds me one of my favorite articles

https://blog.codinghorror.com/the-infinite-space-between-wor...

Re: Cloning a Laptop over NVMe TCP

#147

Post by harikb »

Since there is no mention of FDT (Fast Data Transfer) [1]

Amazing software (transfer performance wise) written (unfortunately in) Java :) Unintuitive CLI options, but fastest transfer I have seen.

So fast, if I don't artificially limit the speed, my command would sometimes hog the entire local network.

-limit Restrict the transfer speed at the specified rate. K (KiloBytes/s), M (MegaBytes/s) or G (GigaBytes/s) may be used as suffixes.

[1] http://monalisa.cern.ch/FDT/download.html

PS: It does cause file-fragmentation on the destination side, but it shouldn't matter to anyone, practically.

Re: Cloning a Laptop over NVMe TCP

#148

Post by M_bara »

mrb wrote: post #5

In the author's scenario, there are zero benefits in using NVMe/TCP, as he just ends up doing a serial block copy using dd(1) so he's not leveraging concurrent I/O. All the complex commands can be replaced by a simple netcat. On the destination laptop: $ nc -l -p 1234 | dd of=/dev/nvme0nX bs=1M On the source laptop: $ nc x.x.x.x 1234 The dd on the destination is just to buffer writes so they are faster/more efficient…

About 9 years ago, I consulted for a company that had a bad internal hack - disgruntled cofounder. Basically, a dead man’s switch was left that copied out the first 20mb of every disk to some bucket and then zeroed out. To recover the data we had to use test disk to rebuilt the partition table… but before doing that we didn’t want to touch the corrupted disks so we ended up copying out about 40tb using rescue flash disks, nectat and drive (some of the servers had a physical raid with all slots occupied so you couldn’t use some free had slots). Something along the lines of dd if=/dev/sdc bs=xxx | gzip | nc -l -p 8888 and the reverse on the other side. It actually worked surprisingly well. One thing of note,try combinations of dd bs to match with sector size - proper sizing had a large impact on dd throughput

Re: Cloning a Laptop over NVMe TCP

#149

Post by mrb »

ufocia wrote: post #97
mrb wrote: post #52

Earlier quoted context omitted.

Your example proves me right. Your drive should be capable of 1000 MB/s but O_DIRECT reduces performance to 400 MB/s. This matters in the specific use case of "netcat | gunzip | dd" as the compressed data rate on GigE will indeed be around 120 MB/s but when gunzip is decompressing unused parts of the filesystem (which compress very well), it will attempt to write 1+ GB/s or more to the pipe to dd and it would not be…

I wonder how using tee to compute the hash in parallel would affect the overall performance.

On GigE or even 2.5G it shouldn't slow things down, as "sha1sum" on my 4-year-old CPU can process at ~400 MB/s (~3.2 Gbit/s). But I don't bother to use tee to compute the hash in parallel because after the disk image has been written to the destination machine, I like to re-read from the destination disk to verify the data was written with integrity. So after the copy I will run sha1sum /dev/XXX on the destination machine. And while I wait for this command to complete I might as well run the same command on the source machine, in parallel. Both commands complete in about the same time so you would not be saving wall clock time.

Fun fact: "openssl sha1" on a typical x86-64 machine is actually about twice faster than "sha1sum" because their code is more optimized.

Another reason I don't bother to use tee to compute the hash in parallel is that it writes with a pretty small block size by default (8 kB) so for best performance you don't want to pass /dev/nvme0nX as the argument to tee, instead you would want to use fancy >(...) shell syntax to pass a file descriptor as an argument to tee which is sha1sum's stdin, then pipe the data to dd to give it the opportunity to buffer writes in 1MB block to the nvme disk:

  $ nc -l -p 1234 | tee >(sha1sum >s.txt) | dd bs=1M of=/dev/XXX
But rescue disks sometimes have a basic shell that doesn't support fancy >(...) syntax. So in the spirit of keeping things simple I don't use tee.

Re: Cloning a Laptop over NVMe TCP

#150

Post by bpye »

wolrah wrote: post #131
crote wrote: post #105

Earlier quoted context omitted.

I doubt it makes a difference. SSDs are an awful lot better at sequential writes than random writes, and concurrent IO would mainly speed up random access. Besides, I don't think anyone really has a local network which is faster than their SSD. Even a 4-year-old consumer Samsung 970 Pro can sustain full-disk writes at 2.000M Byte/s, easily saturating a 10Gbit connection. If we're looking at state-of-the-art consumer…

> Besides, I don't think anyone really has a local network which is faster than their SSD. Even a 4-year-old consumer Samsung 970 Pro can sustain full-disk writes at 2.000M Byte/s, easily saturating a 10Gbit connection. You might be surprised if you take a look at how cheap high speed NICs are on the used market. 25G and 40G can be had for around $50, and 100G around $100. If you need switches things start to get exp…

Depending on your sensitivity to power/noise a 40Gb switch can be had somewhat inexpensively too - something like the Brocade ICX6610 costs <200$ on eBay.
Post reply on HN