121–130 of 181 posts
I love to see such progress, brought by nvme-over-tcp and systemd! Not so many years ago doing something similar (exporting a block device over network, and mounting it from another host) would have meant messing with iscsi which is a very cumbersome thing to do (and quite hard to master).
I recently had to copy around 200gb of files over wifi. I used rsync to make sure a connection failure doesn't mean I have to start over and so that nothing is lost, but it took at least 6 hours. I wonder what could I have done better. Btw, what kinds of guarantees do you get with the dd method? Do you have to compare md5s of the resulting block level devices after?
> I wonder what could I have done better. 6 hours is roughly 10MB/s, so you likely could have gone much much quicker. Did you compress with `-z`? If you could use ethernetyou probably could have done it at closer to 100MB/s on most deviceds, which would have been 35 minutes.
One thing I could have done is found a way to track total progress, so that I could have noticed that this is going way too slow.
I recently had to copy around 200gb of files over wifi. I used rsync to make sure a connection failure doesn't mean I have to start over and so that nothing is lost, but it took at least 6 hours. I wonder what could I have done better. Btw, what kinds of guarantees do you get with the dd method? Do you have to compare md5s of the resulting block level devices after?
dd doesn't skip empty blocks, like clonezilla would do.
I recently had to copy around 200gb of files over wifi. I used rsync to make sure a connection failure doesn't mean I have to start over and so that nothing is lost, but it took at least 6 hours. I wonder what could I have done better. Btw, what kinds of guarantees do you get with the dd method? Do you have to compare md5s of the resulting block level devices after?
If your transport method for rsync was ssh, that is often a bottleneck, as openssh has historically had some weird performance limits that needed obscure patches to get around. Enabling compression helps too if your CPU doesn't become a bottleneck
Earlier quoted context omitted.
Right, that plan is somewhat foiled by most laptops not having ethernet ports anymore. You don't need crossover cables anymore. You can just connect a regular patch cable directly between 2 devices. Modern devices can swap RX/TX as needed. As for auto-configuration, that's up to the OS, but yeah you probably have to set up static IPs.
They would receive APIPA (169.254.0.0/16) address for IPv4 and link-local for IPv6, if the interface would be brought up. Now, that second part is a question; windows would do it, but linux probably not.
I was under the impression you have to set up the devices as each other's default gateway, but maybe I'm the one not up to modern standards this time.
Earlier quoted context omitted.
> I wonder what could I have done better. 6 hours is roughly 10MB/s, so you likely could have gone much much quicker. Did you compress with `-z`? If you could use ethernetyou probably could have done it at closer to 100MB/s on most deviceds, which would have been 35 minutes.
No, I didn't use compression. Would it be useful over a high-bandwidth connection? I presume that it wasn't wifi bandwidth that was bottlenecking, though I've not really checked. One thing I could have done is found a way to track total progress, so that I could have noticed that this is going way too slow.
Buy a USB to Ethernet adapter. It will come in handy in the future.
I recently had to copy around 200gb of files over wifi. I used rsync to make sure a connection failure doesn't mean I have to start over and so that nothing is lost, but it took at least 6 hours. I wonder what could I have done better. Btw, what kinds of guarantees do you get with the dd method? Do you have to compare md5s of the resulting block level devices after?
> ...but it took at least 6 hours... Rsync cannot transfer more than one file at a time so if you were transferring a lot of small files that was probably the bottleneck. You could either use xargs/parallel to split the file list and run multiple instances of rsync or use something like rclone, which supports parallel transfers on its own.
In the author's scenario, there are zero benefits in using NVMe/TCP, as he just ends up doing a serial block copy using dd(1) so he's not leveraging concurrent I/O. All the complex commands can be replaced by a simple netcat. On the destination laptop: $ nc -l -p 1234 | dd of=/dev/nvme0nX bs=1M On the source laptop: $ nc x.x.x.x 1234 The dd on the destination is just to buffer writes so they are faster/more efficient…
I recently had to set up a new laptop (xubuntu). Previously I cloned but I this time I wanted to refresh some of the configs. Using a usb-c cable to transfer at 10gb/s is so useful (as my only other option was WiFi). When you plug the computers together they form an ad-hoc network and you can just rsync across. As far as I could tell the link was saturated so using anything else (other protocols) would be pointless.…
The transfer itself screamed and I had a terabyte over in a few mins. Also I didn't bother with encryption on this one, so that simplified things a lot.