Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloning a Laptop over NVMe TCP

copyninja.in

Post reply on HN
Print view View original

171–180 of 181 posts

Re: Cloning a Laptop over NVMe TCP

#171

Post by dmos62 »

downrightmike wrote: post #143
dmos62 wrote: post #23

I recently had to copy around 200gb of files over wifi. I used rsync to make sure a connection failure doesn't mean I have to start over and so that nothing is lost, but it took at least 6 hours. I wonder what could I have done better. Btw, what kinds of guarantees do you get with the dd method? Do you have to compare md5s of the resulting block level devices after?

Wifi shares the medium (air) with all other radios. It has a random time it waits after it stops if it sees a collision. "Carrier-sense multiple access with collision avoidance (CSMA/CA) in computer networking, is a network multiple access method in which carrier sensing is used, but nodes attempt to avoid collisions by beginning transmission only after the channel is sensed to be "idle".[1][2] When they do transmit,…

I'm not sure I get your meaning. Are you implying that dd over netcat is unreliable because CSMA/CA is unreliable?

Re: Cloning a Laptop over NVMe TCP

#172

Post by mixmastamyk »

dmos62 wrote: post #170
mixmastamyk wrote: post #133

Earlier quoted context omitted.

One of the options of rsync is to print out transfer speed, --progress or verbose or similar.

Yeah, I did that, but that shows stats per file, which is minimally useful. I would see it go up to 15 mb/s for bigger files, but turns out the aggregate speed was less than 10 mb/s.

Right, I guess it doesn't show the total speed until the end, which is probably too late. Although 15MB/s is not very fast these days either.

Elsewhere I recommended the Thunderbolt4 cable—it screamed at ~10GBs and didn't need any optimization like tar/compression.

Re: Cloning a Laptop over NVMe TCP

#173

Post by downrightmike »

dmos62 wrote: post #171
downrightmike wrote: post #143

Earlier quoted context omitted.

Wifi shares the medium (air) with all other radios. It has a random time it waits after it stops if it sees a collision. "Carrier-sense multiple access with collision avoidance (CSMA/CA) in computer networking, is a network multiple access method in which carrier sensing is used, but nodes attempt to avoid collisions by beginning transmission only after the channel is sensed to be "idle".[1][2] When they do transmit,…

I'm not sure I get your meaning. Are you implying that dd over netcat is unreliable because CSMA/CA is unreliable?

Sorry, just that it isn't anywhere near as efficient. There's a lot of random wait that will slow everything down.

Re: Cloning a Laptop over NVMe TCP

#174

Post by asgrdz »

wolrah wrote: post #131
crote wrote: post #105

Earlier quoted context omitted.

I doubt it makes a difference. SSDs are an awful lot better at sequential writes than random writes, and concurrent IO would mainly speed up random access. Besides, I don't think anyone really has a local network which is faster than their SSD. Even a 4-year-old consumer Samsung 970 Pro can sustain full-disk writes at 2.000M Byte/s, easily saturating a 10Gbit connection. If we're looking at state-of-the-art consumer…

> Besides, I don't think anyone really has a local network which is faster than their SSD. Even a 4-year-old consumer Samsung 970 Pro can sustain full-disk writes at 2.000M Byte/s, easily saturating a 10Gbit connection. You might be surprised if you take a look at how cheap high speed NICs are on the used market. 25G and 40G can be had for around $50, and 100G around $100. If you need switches things start to get exp…

This is exactly what I'm looking for. Would you mind sharing what specific 40G card(s) you're using?

Re: Cloning a Laptop over NVMe TCP

#175

Post by fsckboy »

tripflag wrote: post #43
adrian_b wrote: post #36

Earlier quoted context omitted.

According to the documentation of dd, "iflag=fullblock" is required only when dd is used with the "count=" option. Otherwise, i.e. when dd has to read the entire input file because there is no "count=" option, "iflag=fullblock" does not have any documented effect. From "info dd": "If short reads occur, as could be the case when reading from a pipe for example, ‘iflag=fullblock’ ensures that ‘count=’ counts complete i…

Thank you for the correction -- it is likely that I did use count= when I ran into this some 10 years ago (and been paranoid about ever since). I thought a chunk of data was missing in the middle of the output file, causing everything after that to be shifted over, but I'm probably misremembering.

thank you for bringing it up! i wasn't even aware of this potential problem, and I use bs= count= and skip= seek= (sk"i"p means "input") through pipes across the net aaaallll the time for decades.

it pretty much seems iflag=fullblock is a requirement if you want the counts to work, even though the failure times might be rare

Re: Cloning a Laptop over NVMe TCP

#176

Post by dmos62 »

downrightmike wrote: post #173
dmos62 wrote: post #171

Earlier quoted context omitted.

I'm not sure I get your meaning. Are you implying that dd over netcat is unreliable because CSMA/CA is unreliable?

Sorry, just that it isn't anywhere near as efficient. There's a lot of random wait that will slow everything down.

Oh ok, thanks for the insight!

Re: Cloning a Laptop over NVMe TCP

#177

Post by brokenmachine »

verticalscaler wrote: post #96
throwawaaarrgh wrote: post #85

Earlier quoted context omitted.

This is actually much better because nbdcopy can handle sparse files, you can set the number of connections and threads to number of cores, you can force a flush before exit, and enable a progress bar. For unencrypted drives it also supports TLS.

If you really want a progress bar chuck in a 'pv' somewhere into the command posted at the top of the thread.

And "dd conv=sparse" to fix the other problem.

Re: Cloning a Laptop over NVMe TCP

#178

Post by stemlord »

1970-01-01 wrote: post #104
jurgenkesker wrote: post #76

Or just use Clonezilla? Then you also copy only the actual data blocks, and it can autoresize your partitions as well. That's how I always do it. True, I do always just take the NVME disk out of the laptop and put it in a highspeed dock.

Clonezilla is great. It's got one job and it usually succeeds the first time. My only complaint is the initial learning curve requires tinkering. It's still not at the trust level of fire and forget. Experimenting is recommended as backup is never the same thing as a backup and restore and even Clonezilla will have issues recreating partitions on disks that are very different from their source.

Spot on re: learning curve, it absolutely requires a test run or two before you'll get it right

Re: Cloning a Laptop over NVMe TCP

#179

Post by wolrah »

asgrdz wrote: post #174
wolrah wrote: post #131

Earlier quoted context omitted.

> Besides, I don't think anyone really has a local network which is faster than their SSD. Even a 4-year-old consumer Samsung 970 Pro can sustain full-disk writes at 2.000M Byte/s, easily saturating a 10Gbit connection. You might be surprised if you take a look at how cheap high speed NICs are on the used market. 25G and 40G can be had for around $50, and 100G around $100. If you need switches things start to get exp…

This is exactly what I'm looking for. Would you mind sharing what specific 40G card(s) you're using?

I'm using Mellanox ConnectX-3 cards, IIRC they're HP branded. They shipped in Infiniband mode and required a small amount of command line fiddling to put them in ethernet mode but it was pretty close to trivial.

They're PCIe 3.0 x8 cards so they can't max out both ports, but realistically no one who's considering cheap high speed NICs cares about maxing out more than one port.

Re: Cloning a Laptop over NVMe TCP

#180

Post by crote »

Arnavion wrote: post #157
crote wrote: post #109

Earlier quoted context omitted.

No, it wouldn't imply the machine is 10+ years old. Even a state-of-the-art motherboard like the Gigabyte Z790 D AX (which became available in my country today ) has more USB 3 gen1 (5Gbps) ports than gen2 (10Gbps). The 5Gbps ports are just marketed as "USB 3.1" instead of "USB 3.0" these days, because USB naming is confusing and the important part is the "gen x".

To be clear for everyone: USB 3.0, USB 3.1 gen 1, and USB 3.2 gen 1x1 are all names for the same thing, the 5Gbps speed. USB 3.1 gen 2 and USB 3.2 gen 2x1 are both names for the same thing, the 10Gbps speed. USB 3.2 gen 2x2 is the 20Gbps speed. The 3.0 / 3.1 / 3.2 are the version number of the USB specification. The 3.0 version only defined the 5Gbps speed. The 3.1 version added a 10Gbps speed, called it gen 2, and r…

> There's also a 3.2 gen 1x2 10Gbps speed but I've never seen it used.

It's not intended to be used by-design. Basically, it's a fallback for when a gen2x2 link fails to operate at 20Gbps speeds.

Post reply on HN