SSH hacks – a little sanity for remote workers

At my previous customer, we had to SSH through a bounce gateway (SSH key auth), then a bastion host (LDAP password auth), then the target host (LDAP password auth). Since it was quite annoying, I used multiple ssh_config tricks to make it work without having a 1000 lines SSH config, and I wrote a doc to share best practices. I anonymized it and posted it below.

----------------------

ssh_config_best_practices.md

  CanonicalizeHostname yes

  ##############
  ### GitHub ###
  ##############

  Host github.com
    User jdoe
    IdentityFile ~/.ssh/id_rsa_github

  ##################
  ### My Company ###
  ##################

  Host myproject-dev-*
    ProxyJump bastion-dev

  Host myproject-prod-*
    ProxyJump bastion-prod

  Host bastion-dev
    HostName bastion.myproject-dev.mycompany.com
    ProxyJump bounce.myproject-dev.mycompany.com

  Host bastion-prod
    HostName bastion.myproject-prod.mycompany.com
    ProxyJump bounce.myproject-prod.mycompany.com

  Host *.mycompany.com myproject-dev-* myproject-prod-*
    User john_doe
    IdentityFile ~/.ssh/id_rsa_mycompany


  ##############
  ### Common ###
  ##############

  Host *
    ControlMaster auto
    ControlPath ~/.ssh/sockets/%r@%h
    ControlPersist 2h

    # On OS X, UseKeyChain specifies that we should store passphrases in the Keychain.
    IgnoreUnknown UseKeychain
    UseKeychain yes
    AddKeysToAgent yes

- "CanonicalizeHostname" ensures the config is re-parsed after hostname canonicalization. This means that when you SSH into "bastion-dev", SSH re-parses the config using the full hostname "bastion.myproject-dev.mycompany.com", which then correctly matches the entry "Host * .mycompany.com".

- "ProxyJump" was added in OpenSSH 7.2 (2016) and is simpler and more powerful than "ProxyCommand".

- "bastion-xxx" hosts are the only ones whose hostname can be resolved from the bounce gateways. To connect to other hosts, the trick we use in this config is to do two ProxyJumps: your machine --> bounce --> bastion --> target host.

- "ControlMaster" lets you do SSH multiplexing, which in our case is particularly useful when channeling multiple connections through a bastion host. It also persists SSH connections for a while after we disconnect, which speeds up future connections, and avoids typing the password all the time.

- When you ssh into a host, you must enter your LDAP password twice: first for the bastion, then for the target host. If you then ssh into a second host, you must enter your LDAP password only once, since ControlMaster reuses the SSH connection previously established to the bastion. Also, if you close those SSH shells, the connections will persist for two hours (see ControlPersist), so you won't need to type your password for those two hosts if you try to SSH into them again in the next two hours.

- Using this ssh_config, there is no need to add an Host entry for each host. It is not even needed to specify the IP addresses, since they will be resolved using the DNS on the bastion host.

- With this configuration, you can easily copy a file using scp between your local machine and the target host, without needing to first copy it to the bastion, then ssh to the bastion, then copy it to the target host, then remove it from the bastion...

PS: an ssh_config is parsed from top to bottom, so specific comes first, generic comes last. That's why "Host *" must be at the bottom.

Using outbound SSH when that is allowed is compliance with cybersecurity policy.

> blocking outbound SSH would also mean you couldn't use Git, manage servers in the cloud or other locations, transfer files, etc.

That's right, and that's how it is with my current employer.

If you need outbound SSH to work with Git, that probaly means you're working on a side project, not work. Fetching public code needed for work from a hosting site can be done over https.

Managing servers in the cloud, ditto. If managing servers in the cloud isn't part of your job description, why would your workplace enable that?

I think rsync has obsoleted a lot of that:

  rsync -av --progress indir/ x:outdir/

if rsync is available it's easy:

  rsync -av lcldir/ x:remotedir/

  rsync -av x:remotedir/ lcldir/

I've gotten pretty strict about trailing slashes otherwise I end up creating directories within directories depending on whether things exist or not