Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

High Performance SSH/SCP

psc.edu

Post reply on HN
Print view View original

51–60 of 75 posts

Re: High Performance SSH/SCP

#51

Post by rapier1 »

suprjami wrote: post #6
josephg wrote: post #2

Any chance this work can be upstreamed into mainline SSH? I'd love to have better performance for SSH, but I'm probably not going to install and remember to use this just for the few times it would be relevant.

Unlikely. These patches have been carried out-of-tree for over a decade precisely because upstream OpenSSH won't accept them.

More than 2 decades at this point. The primary reasons is that the full patch set would be a burden for them to integrate and they don't prioritize performance for bulk data transfers. Which is perfectly understandable from their perspective. HPN-SSH builds on the expertise of OpenSSH and we follow their work closely - when they make a new release we incorporate it and follow with our own release inside of a week or two (depending on how long the code review and functionality/regression testing takes). We focus on throughput performance which involves receive buffer normalization, private key cipher speed, code optimization, and so forth. We tend to stay clear of anything involve authentication and we never roll our own when it comes to the ciphers.

Re: High Performance SSH/SCP

#52

Post by Veserv »

riobard wrote: post #45
nickcw wrote: post #40

The fact that sftp is not the fastest protocol is well known by rclone users. The main problem is that it packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream. You can only have so many packets outstanding in the standard SFTP implementation (64 is the default) and the buffers are quite small (32k by default) which gives a total outstanding data of 2MB. The highe…

"(sftp) packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream." why is it designed this way? what problems it's supposed to solve?

Because that is a poor characterization of the problem.

It just has a in-flight message/queue limit like basically every other communication protocol. You can only buffer so many messages and space for responses until you run out of space. The problem there is just that the default amount of buffering is very low and is not adaptive to the available space/bandwidth.

Re: High Performance SSH/SCP

#53

Post by nickcw »

riobard wrote: post #45
nickcw wrote: post #40

The fact that sftp is not the fastest protocol is well known by rclone users. The main problem is that it packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream. You can only have so many packets outstanding in the standard SFTP implementation (64 is the default) and the buffers are quite small (32k by default) which gives a total outstanding data of 2MB. The highe…

"(sftp) packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream." why is it designed this way? what problems it's supposed to solve?

Here is some speculation:

SFTP was designed as a remote file system system access protocol rather than transfer a single file like scp.

I suspect that the root of the problem is that SFTP works over a single SSH channel. SSH connections can have multiple channels but usually the server binds a single channel to a single executable so it makes sense to use only a single channel.

Everything flows from that decision - packetisation becomes necessary otherwise you have to wait for all the files to transfer before you can do anything else (eg list a directory) and that is no good for your remote filesystem access.

Perhaps the packets could have been streamed but the way it works is more like an RPC protocol with requests and responses. Each request has a serial number which is copied to the response. This means the client can have many requests in-flight.

There was a proposal for rclone to use scp for the data connections. So we'd use sftp for the day to day file listings, creating directories etc, but do actual file transfers with scp. Scp uses one SSH channel per file so doesn't suffer from the same problems as sftp. I think we abandoned that idea though as many sftp servers aren't configured with scp as well. Also modern versions of OpenSSH (OpenSSH 9.0 released April 2022) use SFTP instead of scp anyway. This was done to fix various vulnerabilities in scp as I understand.

Re: High Performance SSH/SCP

#54

Post by adolph »

nickcw wrote: post #40

The fact that sftp is not the fastest protocol is well known by rclone users. The main problem is that it packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream. You can only have so many packets outstanding in the standard SFTP implementation (64 is the default) and the buffers are quite small (32k by default) which gives a total outstanding data of 2MB. The highe…

> The fastest protocols are the TLS/HTTP based ones which stream data.

I think maybe you are referring to QUIC [0]? It'd be interesting to see some userspace clients/servers for QUIC that compete with Aspera's FASP [1] and operate on a point to point basis like scp. Both use UDP to decrease the overhead of TCP.

0. https://en.wikipedia.org/wiki/QUIC

1. https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol

Re: High Performance SSH/SCP

#55

Post by rapier1 »

Veserv wrote: post #52
riobard wrote: post #45

Earlier quoted context omitted.

"(sftp) packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream." why is it designed this way? what problems it's supposed to solve?

Because that is a poor characterization of the problem. It just has a in-flight message/queue limit like basically every other communication protocol. You can only buffer so many messages and space for responses until you run out of space. The problem there is just that the default amount of buffering is very low and is not adaptive to the available space/bandwidth.

Yeah, it's an issue because there is also the per channel application layer flow control. So when you are using SFTP you have the TCP flow control, the SSH layer flow control, and then the SFTP flow control. The maximum receive buffer ends up being the minimum of all three. HPN-SSH (I'm the dev) normalizes the SSH layer flow control to the TCP receive buffer but we haven't done enough work on SFTP except to bump up the buffer size/outstanding requests. I need to determine if this is effective enough or if I need some dynamism in there as well.

Re: High Performance SSH/SCP

#56

Post by rapier1 »

adolph wrote: post #54
nickcw wrote: post #40

The fact that sftp is not the fastest protocol is well known by rclone users. The main problem is that it packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream. You can only have so many packets outstanding in the standard SFTP implementation (64 is the default) and the buffers are quite small (32k by default) which gives a total outstanding data of 2MB. The highe…

> The fastest protocols are the TLS/HTTP based ones which stream data. I think maybe you are referring to QUIC [0]? It'd be interesting to see some userspace clients/servers for QUIC that compete with Aspera's FASP [1] and operate on a point to point basis like scp. Both use UDP to decrease the overhead of TCP. 0. https://en.wikipedia.org/wiki/QUIC 1. https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol

We've been looking at using QUIC as the transport layer in HPN-SSH. It's more of a pain that you might think because it breaks the SSH authentication paradigm and requires QUIC layer encryption - so a naive implementation would end up encrypting the data twice. I don't want to do that. Mostly what we are thinking about doing is changing the channel multiplexing for bulk data transfers in order to avoid the overhead and buffer issues. If we can rely entirely on TCP for that then we should get even better performance.

Re: High Performance SSH/SCP

#57

Post by rapier1 »

joecool1029 wrote: post #35

This has been around for years (like at least mid-2000’s). Gentoo used to have this patchset available as a USE flag on net-misc/openssh, but some time ago it was moved to net-misc/openssh-contrib (also configurable by useflag). There are some minor usability bugs and I think both endpoints need to have it installed to take advantage. I remember asking ages ago why it wasn’t upstreamed, there were reasons…

to be honest, there was a period of time in about 2010 or 2012 where I simply wasn't maintaining it as well as I should have been. I wouldn't have upstreamed it then either. That's changed a lot since then.

As an aside - you only really need HPN-SSH on the receiving side of the bulk data to get the buffer normalization performance benefits. It turns out the bottleneck is almost entirely on the receiver and the client will send out data as quickly as you like. At least it was like that until OpenSSH 8.8. At that point changes were made where the client would crash if the send buffer exceeded 16MB. So we had to limit OpenSSH to HPN-SSH flows to a maximum of 16MB receive space. Which is annoying but that's still going to be a win for a lot of users.

Re: High Performance SSH/SCP

#58

Post by charonn0 »

riobard wrote: post #45
nickcw wrote: post #40

The fact that sftp is not the fastest protocol is well known by rclone users. The main problem is that it packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream. You can only have so many packets outstanding in the standard SFTP implementation (64 is the default) and the buffers are quite small (32k by default) which gives a total outstanding data of 2MB. The highe…

"(sftp) packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream." why is it designed this way? what problems it's supposed to solve?

Notably, the SFTP specification was never completed. We're working off of draft specs, and presumably these issues wouldn't have made it into a final version.

Re: High Performance SSH/SCP

#59

Post by formerly_proven »

nickcw wrote: post #40

The fact that sftp is not the fastest protocol is well known by rclone users. The main problem is that it packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream. You can only have so many packets outstanding in the standard SFTP implementation (64 is the default) and the buffers are quite small (32k by default) which gives a total outstanding data of 2MB. The highe…

Besides limiting the length and number of outstanding IO requests, SFTP also rides on top of SSH, which also has a limited window size.

Re: High Performance SSH/SCP

#60

Post by adolph »

rapier1 wrote: post #56
adolph wrote: post #54

Earlier quoted context omitted.

> The fastest protocols are the TLS/HTTP based ones which stream data. I think maybe you are referring to QUIC [0]? It'd be interesting to see some userspace clients/servers for QUIC that compete with Aspera's FASP [1] and operate on a point to point basis like scp. Both use UDP to decrease the overhead of TCP. 0. https://en.wikipedia.org/wiki/QUIC 1. https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol

We've been looking at using QUIC as the transport layer in HPN-SSH. It's more of a pain that you might think because it breaks the SSH authentication paradigm and requires QUIC layer encryption - so a naive implementation would end up encrypting the data twice. I don't want to do that. Mostly what we are thinking about doing is changing the channel multiplexing for bulk data transfers in order to avoid the overhead a…

Yeah, my naive implementation thought experiment was oriented towards a side channel brokered by the ssh connection using nginx and curl. Something like source opens nginx to share a file and tells sink via ssh to curl the file from source with a particular cert.

However, I observed that curl [0] uses openssl' quic implementation (for one of its experimental implementations). Another backend for curl is Quiche [1] which has client and server components already, has the userspace crypto etc. It's a little confusing to me, but CloudFlare also has a project quiche [2] which is a Rust crate with a CLI to share and consume files.

0. https://curl.se/docs/http3.html

1. https://github.com/google/quiche/tree/main/quiche/quic

2. https://github.com/cloudflare/quiche

Post reply on HN