Earlier quoted context omitted.
> there were a lot of folders that had many thousands of files in them If you ever need to do something like this again, it's often faster to parallelize rsync. One tool that provides this is fpsync: https://www.fpart.org/fpsync/
And you'd probably use the snapshot feature of a filesystem like btrfs or zfs instead of hardlinks for deduplication :-)
Yes and something like btrfs-send or zfs-send is probably faster than fpsync