Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

High Performance SSH/SCP

psc.edu

Post reply on HN
Print view View original

41–50 of 75 posts

Re: High Performance SSH/SCP

#41

Post by joecool1029 »

rapier1 wrote: post #38
josephg wrote: post #2

Any chance this work can be upstreamed into mainline SSH? I'd love to have better performance for SSH, but I'm probably not going to install and remember to use this just for the few times it would be relevant.

I'm the lead developer. I can go into this a bit more when I get from an appointment if people are interested.

I’m interested. Mainly to update the documentation on it for Gentoo, people have asked about it over the years. Also, TIL it appears HN has a sort of account dormancy status it appears you are in.

Re: High Performance SSH/SCP

#42

Post by Bender »

chasil wrote: post #30
Bender wrote: post #11

Earlier quoted context omitted.

I doubt this would ever be accepted upstream. That said if one wants speed play around with lftp [1]. It has a mirror subsystem that can replicate much of rsync functionality in a chroot sftp-only destination and can use multiple TCP/SFTP streams in a batch upload and per-file meaning one can saturate just about any upstream. I have used this for transferring massive postgres backups and then because I am paranoid wh…

The normal answer that I have heard to the performance problems in the conversion from scp to sftp is to use rsync. The design of sftp is such that it cannot exploit "TCP sliding windows" to maximize bandwidth on high-latency connections. Thus, the migration from scp to sftp has involved a performance loss, which is well-known. https://daniel.haxx.se/blog/2010/12/08/making-sftp-transfers... The rsync question is not…

I included LFTP using mirror+sftp in my example as it is the secure way to give less than trusted people access to files and one can work around the lack of sliding windows by spawning as many TCP flows as one wishes with LFTP. I would love to see SFTP evolve to use sliding windows but for now using it in the data-center or over WAN accelerated links is still fast.

Rsync is great when moving files between trusted systems that one has a shell on but the downside is that rsync can not split up files into multiple streams so there is still a limit based on source+dest buffer+rtt and one has to provide people a shell or add some clunky way to prevent a shell by using wrappers unless using native rsync port 873 which is not encrypted. Some people break up jobs on the client side and spawn multiple rsync jobs in the background. It appears that openrsync is still very much work in progress.

SCP is being or has been deprecated but the binaries still exist for now. People will have to hold onto old binaries and should probably static compile them as the linked libraries will likely go away at some point.

Re: High Performance SSH/SCP

#43

Post by Bender »

zenoprax wrote: post #27
Bender wrote: post #11

Earlier quoted context omitted.

I doubt this would ever be accepted upstream. That said if one wants speed play around with lftp [1]. It has a mirror subsystem that can replicate much of rsync functionality in a chroot sftp-only destination and can use multiple TCP/SFTP streams in a batch upload and per-file meaning one can saturate just about any upstream. I have used this for transferring massive postgres backups and then because I am paranoid wh…

Wow, I hadn't heard of this before. You're saying it can "chunk" large files when operating against a remote sftp-subsystem (OpenSSH)? I often find myself needing to move a single large file rather than many smaller ones but TCP overhead and latency will always keep speeds down.

Not every OS or every SSH daemon support byte ranges but most up to date Linux systems and OpenSSH absolutely support it. One should not assume this exists on legacy systems and daemons.

Re: High Performance SSH/SCP

#44

Post by throw0101d »

chasil wrote: post #31
wahern wrote: post #13

Earlier quoted context omitted.

There's nothing inherently slow about UFS2; the theoretical performance profile should be nearly identical to Ext4. For basic filesystem operations UFS2 and Ext4 will often be faster than more modern filesystems. OpenBSD's filesystem operations are slow not because of UFS2, but because they simply haven't been optimized up-and-down the stack the way Ext4 has been Linux or UFS2 on FreeBSD. And of course, OpenBSD's imp…

OpenBSD UFS did have "soft updates" which were some kind of alternative to journaling. I believe that these were recently removed. Perhaps they don't play well with SMP.

McKusick, who wrote the original BSD FFS, also later came up with SU:

* https://en.wikipedia.org/wiki/Soft_updates

Re: High Performance SSH/SCP

#45

Post by riobard »

nickcw wrote: post #40

The fact that sftp is not the fastest protocol is well known by rclone users. The main problem is that it packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream. You can only have so many packets outstanding in the standard SFTP implementation (64 is the default) and the buffers are quite small (32k by default) which gives a total outstanding data of 2MB. The highe…

"(sftp) packetizes the data and waits for responses, effectively re-implementing the TCP window inside a TCP stream."

why is it designed this way? what problems it's supposed to solve?

Re: High Performance SSH/SCP

#46

Post by fanf2 »

Bender wrote: post #43
zenoprax wrote: post #27

Earlier quoted context omitted.

Wow, I hadn't heard of this before. You're saying it can "chunk" large files when operating against a remote sftp-subsystem (OpenSSH)? I often find myself needing to move a single large file rather than many smaller ones but TCP overhead and latency will always keep speeds down.

Not every OS or every SSH daemon support byte ranges but most up to date Linux systems and OpenSSH absolutely support it. One should not assume this exists on legacy systems and daemons.

Byte ranges are the only way to access files over sftp. Look at the read and write requests in https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filex...

Re: High Performance SSH/SCP

#47

Post by chasil »

Bender wrote: post #42
chasil wrote: post #30

Earlier quoted context omitted.

The normal answer that I have heard to the performance problems in the conversion from scp to sftp is to use rsync. The design of sftp is such that it cannot exploit "TCP sliding windows" to maximize bandwidth on high-latency connections. Thus, the migration from scp to sftp has involved a performance loss, which is well-known. https://daniel.haxx.se/blog/2010/12/08/making-sftp-transfers... The rsync question is not…

I included LFTP using mirror+sftp in my example as it is the secure way to give less than trusted people access to files and one can work around the lack of sliding windows by spawning as many TCP flows as one wishes with LFTP. I would love to see SFTP evolve to use sliding windows but for now using it in the data-center or over WAN accelerated links is still fast. Rsync is great when moving files between trusted sys…

The scp program switched to calling sftp as the server in OpenSSH version 8.9, and notably Windows is now running 9.5, so large segments of scp users are now invoking sftp behind the scenes.

If you want to use the historic scp server instead, a command line option is provided to allow this:

"In case of incompatibility, the scp(1) client may be instructed to use the legacy scp/rcp using the -O flag."

https://www.openssh.org/releasenotes.html

The old scp behavior hasn't been removed, but you need to specifically request it. It is not the default.

It would seem to me that an alternate invocation for file transfer could be tested against sftp in high latency situations:

  ssh yourhost 'cat somefile' > somefile
That would be slightly faster than tar, which adds some overhead. Using tar on both sides would allow transfers of special files, soft links, and retain hard links, which neither scp nor sftp will do.

  ssh yourhost 'tar cf - yourdir' | tar xpf -
Windows has also recently added a tar command.

Re: High Performance SSH/SCP

#48

Post by rapier1 »

chasil wrote: post #30
Bender wrote: post #11

Earlier quoted context omitted.

I doubt this would ever be accepted upstream. That said if one wants speed play around with lftp [1]. It has a mirror subsystem that can replicate much of rsync functionality in a chroot sftp-only destination and can use multiple TCP/SFTP streams in a batch upload and per-file meaning one can saturate just about any upstream. I have used this for transferring massive postgres backups and then because I am paranoid wh…

The normal answer that I have heard to the performance problems in the conversion from scp to sftp is to use rsync. The design of sftp is such that it cannot exploit "TCP sliding windows" to maximize bandwidth on high-latency connections. Thus, the migration from scp to sftp has involved a performance loss, which is well-known. https://daniel.haxx.se/blog/2010/12/08/making-sftp-transfers... The rsync question is not…

Rsync commonly uses SSH as the transport layer so it won't necessarily be any faster than SFTP unless you are using the rsync daemon (usually on port 873). However, the rsync daemon won't provide any encryption and I can't suggest using it unless it's on a private network.

Re: High Performance SSH/SCP

#49

Post by Bender »

fanf2 wrote: post #46
Bender wrote: post #43

Earlier quoted context omitted.

Not every OS or every SSH daemon support byte ranges but most up to date Linux systems and OpenSSH absolutely support it. One should not assume this exists on legacy systems and daemons.

Byte ranges are the only way to access files over sftp. Look at the read and write requests in https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filex...

I agree but there are legacy daemons that do not follow the spec. Most here will never see them in their lifetime but I had to deal with it in the financial world. People would be amazed and terrified at all the old non-standard crap that their payroll data is flying across. They just ignore the range and send the entire file. I am happy to not have to deal with that any more.

Re: High Performance SSH/SCP

#50

Post by rapier1 »

joecool1029 wrote: post #41
rapier1 wrote: post #38

Earlier quoted context omitted.

I'm the lead developer. I can go into this a bit more when I get from an appointment if people are interested.

I’m interested. Mainly to update the documentation on it for Gentoo, people have asked about it over the years. Also, TIL it appears HN has a sort of account dormancy status it appears you are in.

For Gentoo I should put you in touch with my co-developer. He's active in Gentoo and has been maintaining a port for it. I'll point him at this conversation. That said, documentation wise, the HPN-README goes into a lot of detail about the HPN-SSH specific changes. I should point out that while HPN-SSH is a fork we follow OpenSSH. Whenever they come out with a new release we come out with one that incorporates their changes - usually we get this out in about a week.
Post reply on HN