61–70 of 181 posts
Not so many years ago doing something similar (exporting a block device over network, and mounting it from another host) would have meant messing with iscsi which is a very cumbersome thing to do (and quite hard to master).
In the author's scenario, there are zero benefits in using NVMe/TCP, as he just ends up doing a serial block copy using dd(1) so he's not leveraging concurrent I/O. All the complex commands can be replaced by a simple netcat. On the destination laptop: $ nc -l -p 1234 | dd of=/dev/nvme0nX bs=1M On the source laptop: $ nc x.x.x.x 1234 The dd on the destination is just to buffer writes so they are faster/more efficient…
This use of dd may cause corruption! You need iflag=fullblock to ensure it doesn't truncate any blocks, and (at the risk of cargo-culting) conv=sync doesn't hurt as well. I prefer to just nc -l -p 1234 > /dev/nvme0nX.
Is there any way to mount a remote volume on Mac OS using NVMe-TCP?
Earlier quoted context omitted.
Your example proves me right. Your drive should be capable of 1000 MB/s but O_DIRECT reduces performance to 400 MB/s. This matters in the specific use case of "netcat | gunzip | dd" as the compressed data rate on GigE will indeed be around 120 MB/s but when gunzip is decompressing unused parts of the filesystem (which compress very well), it will attempt to write 1+ GB/s or more to the pipe to dd and it would not be…
> Your example proves me right. Your drive should be capable of 1000 MB/s but O_DIRECT reduces performance to 400 MB/s. I noted that the bus I connected the device has 500MBps bandwidth theoretical, no? To cite myself: > Target is a Samsung T7 Shield 2TB, with 1050MB/sec sustained write speed. Bus is USB 3.0 with 500MBps top speed (so I can go %50 of drive speeds). Result is 404MBps, which is fair for the bus.
I recently had to copy around 200gb of files over wifi. I used rsync to make sure a connection failure doesn't mean I have to start over and so that nothing is lost, but it took at least 6 hours. I wonder what could I have done better. Btw, what kinds of guarantees do you get with the dd method? Do you have to compare md5s of the resulting block level devices after?
Rsync cannot transfer more than one file at a time so if you were transferring a lot of small files that was probably the bottleneck. You could either use xargs/parallel to split the file list and run multiple instances of rsync or use something like rclone, which supports parallel transfers on its own.
I don't understand why he didn't pipe btrfs through network. Do a btrfs snapshot first, then btrfs send => nc => network => nc => brtfs receive. That way only blocks in use are sent.
There's one caveat though: With btrfs it is not possible to send snapshots recursively, so if he had lots of recursive snapshots (which can happen in Docker/LXD/Incus), it is relatively hard to mirror the same structure in a new disk. I like btrfs, but recursive send/receive is one aspect ZFS is just better.
Yeah there are much simpler approaches. I don't bother with container images but can install a new machine from an Ansible playbook in a few minutes. I do then have to copy user files (Borg restore) but that results in a clean new install.
In the author's scenario, there are zero benefits in using NVMe/TCP, as he just ends up doing a serial block copy using dd(1) so he's not leveraging concurrent I/O. All the complex commands can be replaced by a simple netcat. On the destination laptop: $ nc -l -p 1234 | dd of=/dev/nvme0nX bs=1M On the source laptop: $ nc x.x.x.x 1234 The dd on the destination is just to buffer writes so they are faster/more efficient…
Note that you can increase pipe buffer, I think default maximum size is usually around 1MB. A bit tricky to do from command line, one possible implementation being https://unix.stackexchange.com/a/328364
In the author's scenario, there are zero benefits in using NVMe/TCP, as he just ends up doing a serial block copy using dd(1) so he's not leveraging concurrent I/O. All the complex commands can be replaced by a simple netcat. On the destination laptop: $ nc -l -p 1234 | dd of=/dev/nvme0nX bs=1M On the source laptop: $ nc x.x.x.x 1234 The dd on the destination is just to buffer writes so they are faster/more efficient…