SSH Bastion Host Best Practices

Good writup. One thing I would add for bastions if you wanted to harden them would be to disable session multiplexing if you are using MFA/2FA.

  MaxSessions 1

The default is 10. The plus side of multiplexing is that subsequent connections using the same ssh connection channels are not validated against the authorization mechanisms such as login or 2FA. This reduces friction and speeds up the login process because login is not actually occurring. The trade-off of multiplexing is that all subsequent logins using that ssh connection are not logged nor are they validated with MFA. This means a person phishing your team members can easily hijack their connections without needing a password or 2FA and there are no lastlog entries. SSH Session multiplexing combined with passwordless sudo makes taking over a company trivial even if they have 2FA and strong passwords.

Another risk with a bastion model is port forwarding. As an organization you have to decide what is appropriate for that bastion. Unrestricted forwarding? Restricted? Denied?

  AllowAgentForwarding                    no
  AllowTcpForwarding                      yes
  PermitOpen                              192.168.1.2:22

If this bastion is for a PCI environment then one may want tighter restrictions. If it is for a development environment then maybe less restrictions and just better auditing on each host to enable forensic remediation.

If your bastion is also used for automation to drop files into a staging area, you can limit that automation to file transfers and even limit what it may do with files. This prevents the automation from having a shell or performing port forwarding.

The keys should be outside of the home directories to prevent malicious tools from appending additional authorized_keys into the account. Make use of automation to manage key trusts and add a comment to keys to map them to an internal tracking system like Jira. This assumes your MFA/2FA is excluding specific accounts or groups via PAM and permitting the use of ssh keys with specific groups or accounts.

  AuthorizedKeysFile               /etc/ssh/keys/%u

  Match Group                      sftpusers
        Banner                     /etc/ssh/banner_sftp.txt
        PubkeyAuthentication       yes
        PasswordAuthentication     no
        PermitEmptyPasswords       no
        GatewayPorts               no
        ChrootDirectory            /data/sftphome/%u
        ForceCommand               internal-sftp -l DEBUG1 -f AUTHPRIV -P symlink,hardlink,fsync,rmdir,remove,rename,posix-rename
        AllowTcpForwarding         no
        AllowAgentForwarding       no

-P sets limits on what may not be done in sftp. -p does the inverse and limits what may be done. [1] -l DEBUG1 or VERBOSE will give you syslog entries of what commands were executed on the files. This is useful for audits. Some redundant settings above are also useful to set explicitly for audits.

Another thing mentioned in the article is iptables. In a PCI environment one may want to also have explicit outbound rules using the owner module to limit what users or groups are permitted to ssh out. So if your organization have a group of people allowed to use this host as a bastions, then one could write a rule like

  iptables -I OUTPUT -m owner --gid-owner devops -p tcp --dport 22 -d 192.168.0.0/16 -j ACCEPT

Or specify what CIDR blocks, ports, protocols may be used. You can use REJECT rules after this rule to make it obvious a connection was not allowed so that people do not spend hours debugging. This module is also handy for limiting which daemons may speak to your infrastructure. How strict or liberal the rule is entirely at the needs of your organization.

Lastly I would add that bastions should have as minimal an OS install possible and have SELinux enforcing. Actions denied by SELinux should go to a security operations center after you spend some time tuning out the noise and false positives.

[1] - https://man7.org/linux/man-pages/man8/sftp-server.8.html

Twice I've seen Bastion Hosts compromised. Both times it practically gave the attackers the highest access. In one case it basically hid where the attack came from (compromised logs and all). In another it let them hijack an admin's password by reading his sudo.

IMHE, Bastion Hosts suck.

If you are forced to use one, send logs to a safer one-way storage encrypted and put tampering triggers everywhere you can in the Bastion Host. Also make sure you log outgoing connections. And make sure you can easily match incoming to outgoing.

If you absolutely have to use sudo on the Bastion Host force it to OTP only. Or if absolutely not possible, use 2FA, but this is a risk as something somewhere might not be properly protected and the password will leak. But the better way would be to have the bastion host run on some read-only image and not letting it upgrade or do any admin task at all. Maybe even remove admin users, SSH, the whole lot.

And related, do not have a single account with god-like access to everything. Isolate permissions. This is probably the hardest to get OK'd but it's the classic SPOF where they got you by the balls.

very nice writeup - one of the better ones i have seen. you can go a step further and eliminate open inbound port 22 (make the sshd server 'dark' to the network) with open source solutions like this:

https://ziti.dev/blog/zitifying-ssh/

disclosure: we build SaaS on top of OpenZiti (the open source) so are opinionated in this domain. and, to be clear, the above is just one layer...other layers of security still apply.

People still use Bastion hosts?

I'm trying to grok why they're better than SOCKS5 proxies... Is it because they provide shell access and a larger attack surface? ;-)

Could someone do me a solid and explain best security practices around bastion hosts and vpn?

e.g. - would you still require users connected to the vpn to go through a bastion host? - would you ever run bastion/vpn through the same box? - are there preferred access use cases for each?