Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloning a Laptop over NVMe TCP

copyninja.in

Post reply on HN
Print view View original

161–170 of 181 posts

Re: Cloning a Laptop over NVMe TCP

#161

Post by maccard »

dmos62 wrote: post #122
maccard wrote: post #75

Earlier quoted context omitted.

> I wonder what could I have done better. 6 hours is roughly 10MB/s, so you likely could have gone much much quicker. Did you compress with `-z`? If you could use ethernetyou probably could have done it at closer to 100MB/s on most deviceds, which would have been 35 minutes.

No, I didn't use compression. Would it be useful over a high-bandwidth connection? I presume that it wasn't wifi bandwidth that was bottlenecking, though I've not really checked. One thing I could have done is found a way to track total progress, so that I could have noticed that this is going way too slow.

I wouldn't consider 10MB/s high bandwidth. My home wifi network can comfortably handle 35MB/s, while a wired connection is 110MB/s+

> No, I didn't use compression. Would it be useful over a high-bandwidth connection?

Compression will be useful if you're IO bound as opposed to CPU bound, which at 10MB/s you definitely are.

Re: Cloning a Laptop over NVMe TCP

#162

Post by hooli_gan »

maccard wrote: post #75
dmos62 wrote: post #23

I recently had to copy around 200gb of files over wifi. I used rsync to make sure a connection failure doesn't mean I have to start over and so that nothing is lost, but it took at least 6 hours. I wonder what could I have done better. Btw, what kinds of guarantees do you get with the dd method? Do you have to compare md5s of the resulting block level devices after?

> I wonder what could I have done better. 6 hours is roughly 10MB/s, so you likely could have gone much much quicker. Did you compress with `-z`? If you could use ethernetyou probably could have done it at closer to 100MB/s on most deviceds, which would have been 35 minutes.

Zstd is also great for io bottlenecked transfers with its adapt flag. I don't know if you can use it with rsync though.

> --adapt[=min=#,max=#]: zstd will dynamically adapt compression level to perceived I/O conditions.

Re: Cloning a Laptop over NVMe TCP

#163

Post by akira2501 »

Nexxxeh wrote: post #160
akira2501 wrote: post #158

Earlier quoted context omitted.

You can "make" windows do a lot of things. As a consumer oriented product you shouldn't have to.

You don't if you move from one relatively modern machine to another. My current laptop SATA SSD has moved between... 4 different machines, I think? I'll have to "make" when I move to an NVMe SSD, maybe, but it might largely just work. Time may tell. If you do things the Microsoft way, you just sign into your MS account and your files show up via OneDrive, your apps and games come from the Microsoft store anyway. Ther…

> If you do things the Microsoft way

and happen to live near good bandwidth.

> Like making it progressively more difficult to just use local accounts.

right. because they don't care about your hardware just your payment relationship with them.

> cloning a machine from hardware a decade old onto a brand new laptop may require expertise

you don't see the connection between all these facts? "coping all the work and effort you've collected on your personal machine for a decade _still_ unaccountably requires expertise."

If this were something so unusual that people would rarely want to do it this would be a reasonable point. The fact that this is such an obvious thing people want to do without having to have a monthly subscription first indicates that the product does not cater well to it's selected market segments.

Re: Cloning a Laptop over NVMe TCP

#165

Post by iforgotpassword »

bayindirh wrote: post #34
iforgotpassword wrote: post #32

Earlier quoted context omitted.

I don't see how you can consider the nvme over tcp version less moving parts. dd is installed on every system, and if you don't have nc you can still use ssh and sacrifice a bit of performance. dd if=/dev/foo | ssh dest@bar "cat > /dev/moo"

NVMe over TCP encapsulates and shows me the remote device as is. Just a block device. I just copy that block device with "dd", that's all. It's just a dumb pipe encapsulated with TCP, which is already battle tested enough. Moreover, if I have fatter pipe, I can tune dd for better performance with a single command.

Nvme over TCP encapsulates the entire nvme protocol in TCP, which is way more complex than just sending the raw data. It's the opposite of "a dumb pipe encapsulated in tcp", this is what the netccat approach would be. Heck if you insist on representing the drive as a block device on the remote side you could just as well use NBD, which is just about as many moving parts as nvme over tcp but still a simpler protocol.

Re: Cloning a Laptop over NVMe TCP

#166

Post by [deleted] »

streb-lo wrote: post #94
tarruda wrote: post #66

Earlier quoted context omitted.

That's the first thing I thought when I saw he used btrfs. I use btrfs send/receive all the time via SSH and it works great. He could easily have setup SSH server in GRML live session. There's one caveat though: With btrfs it is not possible to send snapshots recursively, so if he had lots of recursive snapshots (which can happen in Docker/LXD/Incus), it is relatively hard to mirror the same structure in a new disk.…

Yea, there needs to be a `snapshot -r` option or something. I like using subvolumes to manage what gets a snapshot but sometimes you want the full disk.

[deleted]

Re: Cloning a Laptop over NVMe TCP

#169

Post by dmos62 »

maccard wrote: post #161
dmos62 wrote: post #122

Earlier quoted context omitted.

No, I didn't use compression. Would it be useful over a high-bandwidth connection? I presume that it wasn't wifi bandwidth that was bottlenecking, though I've not really checked. One thing I could have done is found a way to track total progress, so that I could have noticed that this is going way too slow.

I wouldn't consider 10MB/s high bandwidth. My home wifi network can comfortably handle 35MB/s, while a wired connection is 110MB/s+ > No, I didn't use compression. Would it be useful over a high-bandwidth connection? Compression will be useful if you're IO bound as opposed to CPU bound, which at 10MB/s you definitely are.

Thanks for the clarification. I presumed that it's the rsync+ssh protocol chatter that was slowing me down, not the wifi bandwidth (which I've not tested separately ). I don't have my head wrapped around the mechanics in play, to be honest.

Re: Cloning a Laptop over NVMe TCP

#170

Post by dmos62 »

mixmastamyk wrote: post #133
dmos62 wrote: post #122

Earlier quoted context omitted.

No, I didn't use compression. Would it be useful over a high-bandwidth connection? I presume that it wasn't wifi bandwidth that was bottlenecking, though I've not really checked. One thing I could have done is found a way to track total progress, so that I could have noticed that this is going way too slow.

One of the options of rsync is to print out transfer speed, --progress or verbose or similar.

Yeah, I did that, but that shows stats per file, which is minimally useful. I would see it go up to 15 mb/s for bigger files, but turns out the aggregate speed was less than 10 mb/s.
Post reply on HN