SSH Emergency Access • Hacker News Board

Re: SSH Emergency Access

#31

Post by kevincox » Sat, Jul 04, 2020, 4:18 PM UTC

marcosdumay wrote: post #21

kevincox wrote: post #16

Earlier quoted context omitted.

I think it depends. I've worked in places that had something like the following setup. - Hardware in datacenters with operators who were not experts on the applications running. - All remote access was done using a short term (~1 day) ssh keys. There was an authentication service to generate these. It was pretty easy to imagine that the authentication service would go down. In this case a selection of people who work…

> All remote access was done using a short term (~1 day) ssh keys. There was an authentication service to generate these. This is weird. Really weird. Did that service use a more secure authentication storage than a password protected key?

Yes, the system used multi-factor auth and could be locked for suspicious activities.

Re: SSH Emergency Access

#32

Post by tashian » Sat, Jul 04, 2020, 4:22 PM UTC

cbb330 wrote: post #17

Why not use certificates as your primary authentication for SSH? Facebook has a great blog post on implementing this at scale: https://engineering.fb.com/security/scalable-and-secure-acce...

Yes! Shameless plug — we (smallstep.com) offer a service that makes this frictionless at scale and super easy to set up. You'll never want to go back to public keys.

Re: SSH Emergency Access

#33

Post by johnklos » Sat, Jul 04, 2020, 4:23 PM UTC

timeattack wrote: post #14

It's cool and interesting application of the technology, but doesn't really seem to be practical. When you're unable to access machine using your standard SSH keys usually it means that it's highly unlikely that it will be possible to login remotely via other means. As an emergency login there are two common options: * in case of cloud: use remote VM console provided by the hosting provider. * in case of bare-metal:…

IPMI is painfully insecure, and therefore assumes the existence of a completely separate, protected network. Some people don't colocate more than a few machines (and therefore can't justify the extra infrastructure for an IPMI OOB network), don't want to pay extra for a colo provider to provide IPMI OOB, and/or don't trust their colo provider to have access to such a sensitive and insecure thing.

Having an emergency method to connect is an excellent idea.

Re: SSH Emergency Access

#34

Post by taftster » Sat, Jul 04, 2020, 4:24 PM UTC

lazyant wrote: post #26

isatty wrote: post #15

Earlier quoted context omitted.

Yep, the most common way I've lost access to machines is by messing up the iptables/ipfw rules. Read a post here about avoiding that by having a timed reset with sleep.

For people asking: you can create a resetfw.sh script, for iptables: #!/bin/bash iptables -P INPUT ACCEPT iptables -P FORWARD ACCEPT iptables -P OUTPUT ACCEPT iptables -t nat -F iptables -t mangle -F iptables -F iptables -X chmod +x resetfw.sh and add it for ex to /etc/cron.hourly directory This way you can test your iptables rules and they'll get clear at every hour. Once you check they are OK you can delete this cr…

Or possibly just turn iptables off, in the same cron.hourly.

Re: SSH Emergency Access

#35

Post by alexandrerond » Sat, Jul 04, 2020, 4:25 PM UTC

I'm very confused, given Yubikeys have smart card fuctionality and they can be used by gpg-agent to SSH with the regular gpg key (you can add to authorized_keys just like any other keys) and you don't have to go through this whole mess of setup to create a CA and install it.

What am I missing?

Re: SSH Emergency Access

#36

Post by benedikt » Sat, Jul 04, 2020, 4:33 PM UTC

lazyant wrote: post #26

isatty wrote: post #15

Earlier quoted context omitted.

Yep, the most common way I've lost access to machines is by messing up the iptables/ipfw rules. Read a post here about avoiding that by having a timed reset with sleep.

For people asking: you can create a resetfw.sh script, for iptables: #!/bin/bash iptables -P INPUT ACCEPT iptables -P FORWARD ACCEPT iptables -P OUTPUT ACCEPT iptables -t nat -F iptables -t mangle -F iptables -F iptables -X chmod +x resetfw.sh and add it for ex to /etc/cron.hourly directory This way you can test your iptables rules and they'll get clear at every hour. Once you check they are OK you can delete this cr…

https://manpages.debian.org/stretch/iptables/iptables-apply....

Or use `at` to run `iptables-restore`. Simpler than setting up a cronjob (and if youre doing it manually, cron has a bunch of gotchas that at least bite me in the ass once in a blue moon).

Re: SSH Emergency Access

#37

Post by munchbunny » Sat, Jul 04, 2020, 4:35 PM UTC

This is a pedantic detail, but if you're trying to implement this system, it does matter: "resident key" is not a required feature here. You're not using the hardware token for its WebAuthn capability, you're using it for its smart card capability.

You just need PKCS11 token support for SSH, which the YubiKey's smart card capability can do. YubiKey 4 and YubiKey FIPS can both do it, and so can regular old smart cards even though that form factor is a lot less popular now.

The workflow is the same: generate a key pair on the hardware token, have the CA sign it, install the signed cert onto the hardware token, and then SSH with it.

Re: SSH Emergency Access

#38

Post by tashian » Sat, Jul 04, 2020, 4:36 PM UTC

alexandrerond wrote: post #35

I'm very confused, given Yubikeys have smart card fuctionality and they can be used by gpg-agent to SSH with the regular gpg key (you can add to authorized_keys just like any other keys) and you don't have to go through this whole mess of setup to create a CA and install it. What am I missing?

That sounds like a great option too, depending on your situation.

One difference is that the CA is on the hardware key, but the cert (and its private key) is not.

Imagine you're on a team of 50, and anyone on the team might need emergency access to a host at some point. You wouldn't want to buy 50 keys and 50 safes. Just designate a couple folks to manage emergency access. They can manually mint a cert for a colleague as needed, and send it over a secure channel. No security key needed to use the cert, and it self-destructs after a few minutes.

Re: SSH Emergency Access

#39

Post by theatrus2 » Sat, Jul 04, 2020, 4:38 PM UTC

cbb330 wrote: post #17

Why not use certificates as your primary authentication for SSH? Facebook has a great blog post on implementing this at scale: https://engineering.fb.com/security/scalable-and-secure-acce...

If you’re on AWS and have credentials for users there, you can also run bless

https://github.com/Netflix/bless

Re: SSH Emergency Access

#40

Post by munchbunny » Sat, Jul 04, 2020, 4:39 PM UTC

alexandrerond wrote: post #35

I'm very confused, given Yubikeys have smart card fuctionality and they can be used by gpg-agent to SSH with the regular gpg key (you can add to authorized_keys just like any other keys) and you don't have to go through this whole mess of setup to create a CA and install it. What am I missing?

It's a chicken and egg problem: if you can't SSH into the machine, how do you add your key to the SSH config on the target machine?

You could use a very long lived key, but then as soon as you have multiple people who might need production SSH access, you've got access control and revocation issues. The SSH CA is a good minimal solution, because the CA can issue only short-lived SSH keys (few hours at a time) that you use once and throw away. Also, CA trust scales better because it moves user management burden to the certificate issuing process and removes the need to modify the SSH config every time you onboard a new user.

It's a pretty standard practice. Here's a post from Facebook about it from several years ago. This post is just about how to do it using YubiKeys. https://engineering.fb.com/security/scalable-and-secure-acce...