Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

High Performance SSH/SCP

psc.edu

Post reply on HN
Print view View original

61–70 of 75 posts

Re: High Performance SSH/SCP

#61

Post by Veserv »

adolph wrote: post #54
nickcw wrote: post #40

The fact that sftp is not the fastest protocol is well known by rclone users. The main problem is that it packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream. You can only have so many packets outstanding in the standard SFTP implementation (64 is the default) and the buffers are quite small (32k by default) which gives a total outstanding data of 2MB. The highe…

> The fastest protocols are the TLS/HTTP based ones which stream data. I think maybe you are referring to QUIC [0]? It'd be interesting to see some userspace clients/servers for QUIC that compete with Aspera's FASP [1] and operate on a point to point basis like scp. Both use UDP to decrease the overhead of TCP. 0. https://en.wikipedia.org/wiki/QUIC 1. https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol

Available QUIC implementations are very slow. MsQUIC is one of the fastest and can only reach a meager ~7 Gb/s [1]. Most commercial implementations sit in the 2-4 Gb/s range.

To be fair, that is not really a problem of the protocol, just the implementations. You can comfortably drive 10x that bandwidth with a reasonable design.

[1] https://microsoft.github.io/msquic/

Re: High Performance SSH/SCP

#62

Post by nickcw »

adolph wrote: post #54
nickcw wrote: post #40

The fact that sftp is not the fastest protocol is well known by rclone users. The main problem is that it packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream. You can only have so many packets outstanding in the standard SFTP implementation (64 is the default) and the buffers are quite small (32k by default) which gives a total outstanding data of 2MB. The highe…

> The fastest protocols are the TLS/HTTP based ones which stream data. I think maybe you are referring to QUIC [0]? It'd be interesting to see some userspace clients/servers for QUIC that compete with Aspera's FASP [1] and operate on a point to point basis like scp. Both use UDP to decrease the overhead of TCP. 0. https://en.wikipedia.org/wiki/QUIC 1. https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol

Actually the fastest ones in my experience are the HTTP/1.x ones. HTTP/2 is generally slower in rclone though I think that is the fault of the stlib not opening more connections. I haven't really tried QUIC

I just think for streaming lots of data quickly HTTP/1.x plus TLS plus TCP has received many more engineering hours of optimization than any other combo.

Re: High Performance SSH/SCP

#63

Post by adrian_b »

nickcw wrote: post #40

The fact that sftp is not the fastest protocol is well known by rclone users. The main problem is that it packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream. You can only have so many packets outstanding in the standard SFTP implementation (64 is the default) and the buffers are quite small (32k by default) which gives a total outstanding data of 2MB. The highe…

When you are limited to use SSH as the transport, you can still do better than using scp or sftp by using rsync with --rsh="ssh ...".

Besides being faster, with rsync and the right command options you can be certain that it makes exact file copies, together with any file metadata, even between different operating systems and file systems.

I have not checked if in recent years all the bugs of scp and sftp have been fixed, but some years ago there were cases when scp and sftp were losing silently, without warnings, some file metadata (e.g. high-precision timestamps, which were truncated, or extended file attributes).

I am using ssh every day, but there are decades since I have last used scp or sftp, with the exception of the cases when I have to connect to a server that I cannot control and where it happens that rsync is not installed. Even on such servers, if I may add an executable in my home directory, I first copy there an rsync with scp, then I do any other copies with that rsync.

Re: High Performance SSH/SCP

#64

Post by rapier1 »

chasil wrote: post #47
Bender wrote: post #42

Earlier quoted context omitted.

I included LFTP using mirror+sftp in my example as it is the secure way to give less than trusted people access to files and one can work around the lack of sliding windows by spawning as many TCP flows as one wishes with LFTP. I would love to see SFTP evolve to use sliding windows but for now using it in the data-center or over WAN accelerated links is still fast. Rsync is great when moving files between trusted sys…

The scp program switched to calling sftp as the server in OpenSSH version 8.9, and notably Windows is now running 9.5, so large segments of scp users are now invoking sftp behind the scenes. If you want to use the historic scp server instead, a command line option is provided to allow this: "In case of incompatibility, the scp(1) client may be instructed to use the legacy scp/rcp using the -O flag." https://www.opens…

Keep in mind that SCP/SSH might be faster in some cases than SFTP but in both cases it is still limited to a 2MB application layer receive window which is drastically undersized in a lot of situations. It doesn't matter what the TCP window is set to because the OpenSSH window overrides that value. Basically, if your bandwidth delay product is more than 2MB (e.g. 1gbps @ 17ms RTT) you're going to be application limited by OpenSSH. HPN-SSH gets most of the performance benefit by normalizing the application layer receive window to the TCP receive window (up to 128MB). In some cases you'll see 100X throughput improvement on well tuned hosts on a high delay path.

If your BDP is less than 2MB you still might get some benefit if you are CPU limited and use the parallel ciphers. However, the fastest cipher is AES-GCM and we haven't parallelized that as of yet (that's next on the list).

Re: High Performance SSH/SCP

#65

Post by rapier1 »

nickcw wrote: post #40

The fact that sftp is not the fastest protocol is well known by rclone users. The main problem is that it packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream. You can only have so many packets outstanding in the standard SFTP implementation (64 is the default) and the buffers are quite small (32k by default) which gives a total outstanding data of 2MB. The highe…

If you want to see the impact that the flow control buffer size has on OpenSSH I put up a graph based on data collected last week. Basically, it has a huge impact on throughput.

https://gist.github.com/rapier1/325de17bbb85f1ce663ccb866ce2...

Re: High Performance SSH/SCP

#66

Post by rapier1 »

joecool1029 wrote: post #39
Zambyte wrote: post #12

Earlier quoted context omitted.

I'm not fear mongering. I'm just saying - IF you don't trust it - AND you want to use it => run it on a private network You don't have to trust it for security to use it. Putting services on secure networks when the public doesn't need access is standard practice.

I remember the last time I really cared to look into this was in the 2000’s, I had these wdtv embedded boxes that had a super anemic cpu that doing local copies with scp was slow as hell from the cipher overhead. I believe at the time it was possible to disable ciphers in scp but it was still slower than smbfs. NFS was to be avoided as wifi was shit then and losing connection meant risking system locking up. This of…

It's still possible but we only suggest doing it on private known secure networks or when it's data you don't care about. Authentication is still fully encrypted - we just rekey post authentication with a null cipher.

Re: High Performance SSH/SCP

#67

Post by adolph »

nickcw wrote: post #62
adolph wrote: post #54

Earlier quoted context omitted.

> The fastest protocols are the TLS/HTTP based ones which stream data. I think maybe you are referring to QUIC [0]? It'd be interesting to see some userspace clients/servers for QUIC that compete with Aspera's FASP [1] and operate on a point to point basis like scp. Both use UDP to decrease the overhead of TCP. 0. https://en.wikipedia.org/wiki/QUIC 1. https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol

Actually the fastest ones in my experience are the HTTP/1.x ones. HTTP/2 is generally slower in rclone though I think that is the fault of the stlib not opening more connections. I haven't really tried QUIC I just think for streaming lots of data quickly HTTP/1.x plus TLS plus TCP has received many more engineering hours of optimization than any other combo.

Maybe this is one of those things where "Worse is Better" [0] given HTTP/1.x will always receive more time/attention/resources than something that might be theoretically superior but never got the resources to fulfill its promise. Cloudflare is probably one of the few organizations outside of Google with an internal economic case to support QUIC. For everyone else there is the option of paying IBM for Aspera using FASP.

0. https://en.wikipedia.org/wiki/Worse_is_better

Re: High Performance SSH/SCP

#68

Post by adolph »

Veserv wrote: post #61
adolph wrote: post #54

Earlier quoted context omitted.

> The fastest protocols are the TLS/HTTP based ones which stream data. I think maybe you are referring to QUIC [0]? It'd be interesting to see some userspace clients/servers for QUIC that compete with Aspera's FASP [1] and operate on a point to point basis like scp. Both use UDP to decrease the overhead of TCP. 0. https://en.wikipedia.org/wiki/QUIC 1. https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol

Available QUIC implementations are very slow. MsQUIC is one of the fastest and can only reach a meager ~7 Gb/s [1]. Most commercial implementations sit in the 2-4 Gb/s range. To be fair, that is not really a problem of the protocol, just the implementations. You can comfortably drive 10x that bandwidth with a reasonable design. [1] https://microsoft.github.io/msquic/

Thank you for linking to those benchmarks. Its interesting that in OpenSSL upload, Windows userspace is 16% faster than Linux.

Re: High Performance SSH/SCP

#69

Post by ectospheno »

chasil wrote: post #30
Bender wrote: post #11

Earlier quoted context omitted.

I doubt this would ever be accepted upstream. That said if one wants speed play around with lftp [1]. It has a mirror subsystem that can replicate much of rsync functionality in a chroot sftp-only destination and can use multiple TCP/SFTP streams in a batch upload and per-file meaning one can saturate just about any upstream. I have used this for transferring massive postgres backups and then because I am paranoid wh…

The normal answer that I have heard to the performance problems in the conversion from scp to sftp is to use rsync. The design of sftp is such that it cannot exploit "TCP sliding windows" to maximize bandwidth on high-latency connections. Thus, the migration from scp to sftp has involved a performance loss, which is well-known. https://daniel.haxx.se/blog/2010/12/08/making-sftp-transfers... The rsync question is not…

> The rsync question is not a workable answer, as OpenBSD has reimplemented the rsync protocol in a new codebase

I thought openrsync existed solely because of rpki. Even OpenBSD devs recommend using the real version from ports.

Re: High Performance SSH/SCP

#70

Post by chasil »

rapier1 wrote: post #64
chasil wrote: post #47

Earlier quoted context omitted.

The scp program switched to calling sftp as the server in OpenSSH version 8.9, and notably Windows is now running 9.5, so large segments of scp users are now invoking sftp behind the scenes. If you want to use the historic scp server instead, a command line option is provided to allow this: "In case of incompatibility, the scp(1) client may be instructed to use the legacy scp/rcp using the -O flag." https://www.opens…

Keep in mind that SCP/SSH might be faster in some cases than SFTP but in both cases it is still limited to a 2MB application layer receive window which is drastically undersized in a lot of situations. It doesn't matter what the TCP window is set to because the OpenSSH window overrides that value. Basically, if your bandwidth delay product is more than 2MB (e.g. 1gbps @ 17ms RTT) you're going to be application limite…

When I need speed, I drop down to FTP/rcp or some other cleartext protocol.

Moving a terabyte database in an upgrade, I have connected three ports direct (no switch), then used xargs to keep all three connections busy with transferring the 2gb data files. I can get the transfer done in under an hour this way.

I don't currently have a performance need for an encrypted transfer, but one may arise.

Post reply on HN