Live data from Hacker News

SSH Bastion Host Best Practices

goteleport.com

41–50 of 91 posts

Re: SSH Bastion Host Best Practices

#41
post #9

I think this article has come up before? Either way, it's a quirky thing for Gravitational to post, since their flagship project --- Teleport --- basically eliminates bastion servers altogether (you might think of it as an API-controlled self-contained bastion server). Teleport is free, and worth checking out: it solves a bunch of SSH management problems, not just controlling access, but also linking SSH access to SS…

[deleted]

Re: SSH Bastion Host Best Practices

#42
post #6

very nice writeup - one of the better ones i have seen. you can go a step further and eliminate open inbound port 22 (make the sshd server 'dark' to the network) with open source solutions like this: https://ziti.dev/blog/zitifying-ssh/ disclosure: we build SaaS on top of OpenZiti (the open source) so are opinionated in this domain. and, to be clear, the above is just one layer...other layers of security still apply.

i generally end up liking what teleport is doing and what they are all about... i keep meaning to try their opensource stuff out. does teleport's sshd 'listen' on port 22 and does it need an opening in a firewall? still, having sshd listen on localhost and not a public ip is pretty cool imo. Ken and I did exactly that on a stream one day https://youtu.be/oSlwZcwZcsU if anyone is interested. The one extra step one cou…

> does teleport's sshd 'listen' on port 22 and does it need an opening in a firewall?

Sorry, one of those crappy it depends answers. The teleport node agents, the agent running on the server you want a session on, can be configured to listen to inbound connections from the proxy (but doesn't use port 22 by default), or can be configured in a reverse tunneling mode where it does outbound dialing towards the Teleport proxy service. When using the reverse tunneling mode, you don't need inbound access to the end nodes, but still need the nodes to be able to make an outward connection to the Teleport infrastructure.

This is how the cloud hosted Teleport works as well, we can't be expected to have outbound network access to peoples machines, so all the agents will dial the cloud hosted proxies, and setup reverse tunnels that are then used for the inbound connection requests.

In most setups though, the Teleport Proxies would then still have inbound connectivity and are meant to be internet facing, so a client can request an SSH or other session, but that single way into the environment can be hardened, layered with additional security, as the environment may require.

Note: I'm affiliated with Teleport, my comments are my own.

Re: SSH Bastion Host Best Practices

#43

Can someone explain to me the benefits of limiting the IPs that can SSH into the bastion? It seems to me the main thing that's protecting against are misconfigurations of SSH (accidentally letting root log in with no password or something) or a zero day in SSH but I'm not convinced by either.

The company I work for does it so that bastions hosted on some public cloud hosting service are only accessible from the company network or by machines connected to its VPN. We handle _very_ sensitive data, and some engineer screwing up the configuration for a bastion would be _very_ bad. Defence in depth is important.

Re: SSH Bastion Host Best Practices

#44
post #9

I think this article has come up before? Either way, it's a quirky thing for Gravitational to post, since their flagship project --- Teleport --- basically eliminates bastion servers altogether (you might think of it as an API-controlled self-contained bastion server). Teleport is free, and worth checking out: it solves a bunch of SSH management problems, not just controlling access, but also linking SSH access to SS…

An interesting category of HOWTO have been companies teaching you how to do it yourself, for real. Before you get started, they pitch you at the end offering a paid-for option that has 0 learning curve. That's a good pitch.

Re: SSH Bastion Host Best Practices

#45
post #21
post #4

Twice I've seen Bastion Hosts compromised. Both times it practically gave the attackers the highest access. In one case it basically hid where the attack came from (compromised logs and all). In another it let them hijack an admin's password by reading his sudo. IMHE, Bastion Hosts suck. If you are forced to use one, send logs to a safer one-way storage encrypted and put tampering triggers everywhere you can in the B…

I can see that you can get a lot of things wrong with a bastion host, but if implemented sensibly, it should just be one more layer of a defense-in-depth strategy. What would you recommend instead of a bastion host?

> What would you recommend instead of a bastion host?

The question isn't to replace, but to remove. If you apply the same security to the actual hosts (which you probably should anyway) then why have an intermediary?

Re: SSH Bastion Host Best Practices

#46
post #7

There's a few weird things, but it's mostly okay. Do not trust the firewall on the bastion host, if an attack can get into the bastion host, they can disable the firewall, so it cannot be used to limit egress. It's better than nothing, but consider using a firewall that's managed on a via a separate management network. I do agree that you should only allow SSH from a few known IPs. Limiting the number of users is wei…

> if an attack can get into the bastion host, they can disable the firewall, so it cannot be used to limit egress. This assumes that the attacker can get unconstrained root access to the system. It's fine to assume that attackers will but it's not as if you can't make that difficult. Agree with the rest of what you said though.

At least in the DoD and IC environments I've worked in that had bastion hosts, the bastion host was severely locked down:

- Shell compiled without built-ins

- No coreutils

- No sudo

- Root account disabled

- Read-only root filesystem

- No user home directories

- Destroyed and rebuilt from template every X hours on some maintenance schedule

Effectively, all you can do is ssh in, ssh out, and forward ports. It might be theoretically possible, but as far as I know, no one has ever compromised one, especially since you can already only get to the bastion from a government VPN anyway, and authentication to that requires a smart card, so there are an awful lot of things you need to compromise to get to that point.

This also answers the suggestion down the page of "why don't you just apply the same controls to every host and not have a bastion." Because the bastion is unusable and you want to actually use your other hosts.

Re: SSH Bastion Host Best Practices

#47
post #32
post #7

There's a few weird things, but it's mostly okay. Do not trust the firewall on the bastion host, if an attack can get into the bastion host, they can disable the firewall, so it cannot be used to limit egress. It's better than nothing, but consider using a firewall that's managed on a via a separate management network. I do agree that you should only allow SSH from a few known IPs. Limiting the number of users is wei…

> A final recommendation: If you use AWS though, consider using Session Manager instead of SSH and drop the bastion host. You can still connect using the SSH command, using proxy command in OpenSSH, but no public IP or bastion host is required. Yes, this. Also check out https://github.com/rewindio/aws-connect for a convenient wrapper around SSM to make it easier to use (I'm not the author).

I wrote something similar after I moved our fleet to SSM because I didn't want yet another CLI app to memorize flags on. It's ruby based and runs in an interactive mode by default. It doesn't cover the whole set of `aws ssm` featureset but focuses just on things that are needed for debugging sort of tasks. Leaving it here incase it's useful to anyone else: https://github.com/ajbdev/ruby-ssm-ops

Re: SSH Bastion Host Best Practices

#48
post #37

It recommends: # Configure idle time logout ClientAliveInterval but i don't think this is correct. AFAICT, this is a keep-alive mechanism, not a timeout. I don't think openssh has an option to kill idle sessions.

Correct. For that you would populate the variable TMOUT to a positive number in seconds and make that variable read-only

  grep ^read /etc/profile.d/timeout.sh
  readonly TMOUT=7200
This variable can also be set in tmux and gnu screen. People usually figure out fairly quick how to bypass the timer but it is handy when people console into servers via the drac/ilo and forget to log out. Some shells don't do anything with TMOUT so a bastion must only have vetted shells.

Re: SSH Bastion Host Best Practices

#49
post #9

I think this article has come up before? Either way, it's a quirky thing for Gravitational to post, since their flagship project --- Teleport --- basically eliminates bastion servers altogether (you might think of it as an API-controlled self-contained bastion server). Teleport is free, and worth checking out: it solves a bunch of SSH management problems, not just controlling access, but also linking SSH access to SS…

An interesting category of HOWTO have been companies teaching you how to do it yourself, for real. Before you get started, they pitch you at the end offering a paid-for option that has 0 learning curve. That's a good pitch.

Yes, McDonald did this kind of thing a few years back https://youtu.be/rcu4Bj3xEyI
Post reply on HN