Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

High Performance SSH/SCP

psc.edu

Post reply on HN
Print view View original

71–75 of 75 posts

Re: High Performance SSH/SCP

#71

Post by imcritic »

adrian_b wrote: post #63
nickcw wrote: post #40

The fact that sftp is not the fastest protocol is well known by rclone users. The main problem is that it packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream. You can only have so many packets outstanding in the standard SFTP implementation (64 is the default) and the buffers are quite small (32k by default) which gives a total outstanding data of 2MB. The highe…

When you are limited to use SSH as the transport, you can still do better than using scp or sftp by using rsync with --rsh="ssh ...". Besides being faster, with rsync and the right command options you can be certain that it makes exact file copies, together with any file metadata, even between different operating systems and file systems. I have not checked if in recent years all the bugs of scp and sftp have been fi…

I have the opposite opinion and experience: a simple file copy is pretty trivial with scp, but with rsync - it's a goddamn lottery. Too many options, too many possible modes and thus I am never sure about the outcome meeting my expectations.

Re: High Performance SSH/SCP

#72

Post by zenoprax »

aidenn0 wrote: post #29
zenoprax wrote: post #27

Earlier quoted context omitted.

Wow, I hadn't heard of this before. You're saying it can "chunk" large files when operating against a remote sftp-subsystem (OpenSSH)? I often find myself needing to move a single large file rather than many smaller ones but TCP overhead and latency will always keep speeds down.

I use lftp a lot because of it's better UI compared to sftp. However, for large files, even with scp I can pin GigE with an old Xeon-D system acting as a server.

Yes, for local access this is my experience too. For trans-oceanic file transfers I can really see the limits and parallelization is essential.

Re: High Performance SSH/SCP

#73

Post by adrian_b »

imcritic wrote: post #71
adrian_b wrote: post #63

Earlier quoted context omitted.

When you are limited to use SSH as the transport, you can still do better than using scp or sftp by using rsync with --rsh="ssh ...". Besides being faster, with rsync and the right command options you can be certain that it makes exact file copies, together with any file metadata, even between different operating systems and file systems. I have not checked if in recent years all the bugs of scp and sftp have been fi…

I have the opposite opinion and experience: a simple file copy is pretty trivial with scp, but with rsync - it's a goddamn lottery. Too many options, too many possible modes and thus I am never sure about the outcome meeting my expectations.

You have to figure the correct options only once, then you should use forever the same alias or script for copying.

For instance, I always use this:

  ALIAS='/usr/bin/rsync --archive --xattrs --acls --hard-links --progress --rsh="ssh -p PORT -l USER"'
The default options of scp, like also those of cp or of any other UNIX copying program are bad, as they do not make exact copies.

In decades of working with computers, I have never wanted to make any other kind of copies except exact copies, so I never use the default options of cp, scp, rsync etc., but I always use the same aliases for them, with the options needed for exact copies.

Re: High Performance SSH/SCP

#74

Post by rapier1 »

chasil wrote: post #70
rapier1 wrote: post #64

Earlier quoted context omitted.

Keep in mind that SCP/SSH might be faster in some cases than SFTP but in both cases it is still limited to a 2MB application layer receive window which is drastically undersized in a lot of situations. It doesn't matter what the TCP window is set to because the OpenSSH window overrides that value. Basically, if your bandwidth delay product is more than 2MB (e.g. 1gbps @ 17ms RTT) you're going to be application limite…

When I need speed, I drop down to FTP/rcp or some other cleartext protocol. Moving a terabyte database in an upgrade, I have connected three ports direct (no switch), then used xargs to keep all three connections busy with transferring the 2gb data files. I can get the transfer done in under an hour this way. I don't currently have a performance need for an encrypted transfer, but one may arise.

I fully understand that. We're using this, along with parsyncfp2 (which you should checkout) to move 1.5PB of data a month across a 40Gb link. Not saying that HPN-SSH is only useful in that context but different people certainly do have different needs.

Re: High Performance SSH/SCP

#75

Post by chasil »

rapier1 wrote: post #74
chasil wrote: post #70

Earlier quoted context omitted.

When I need speed, I drop down to FTP/rcp or some other cleartext protocol. Moving a terabyte database in an upgrade, I have connected three ports direct (no switch), then used xargs to keep all three connections busy with transferring the 2gb data files. I can get the transfer done in under an hour this way. I don't currently have a performance need for an encrypted transfer, but one may arise.

I fully understand that. We're using this, along with parsyncfp2 (which you should checkout) to move 1.5PB of data a month across a 40Gb link. Not saying that HPN-SSH is only useful in that context but different people certainly do have different needs.

If encryption was absolutely required, I might try it over s_client/s_server or two stunnels, one in client=yes mode.

I'm assuming that would have different limits than you outlined above, although I don't think they do multi core.

Post reply on HN