SSH Emergency Access • Hacker News Board

Re: SSH Emergency Access

#21

Post by marcosdumay » Sat, Jul 04, 2020, 3:19 PM UTC

kevincox wrote: post #16

timeattack wrote: post #14

It's cool and interesting application of the technology, but doesn't really seem to be practical. When you're unable to access machine using your standard SSH keys usually it means that it's highly unlikely that it will be possible to login remotely via other means. As an emergency login there are two common options: * in case of cloud: use remote VM console provided by the hosting provider. * in case of bare-metal:…

I think it depends. I've worked in places that had something like the following setup. - Hardware in datacenters with operators who were not experts on the applications running. - All remote access was done using a short term (~1 day) ssh keys. There was an authentication service to generate these. It was pretty easy to imagine that the authentication service would go down. In this case a selection of people who work…

> All remote access was done using a short term (~1 day) ssh keys. There was an authentication service to generate these.

This is weird. Really weird.

Did that service use a more secure authentication storage than a password protected key?

Re: SSH Emergency Access

#22

Post by BillinghamJ » Sat, Jul 04, 2020, 3:24 PM UTC

chrisweekly wrote: post #5

withinboredom wrote: post #4

Earlier quoted context omitted.

You can physically put the (yubikey) device in a vault that will physically sound an alarm when opened. It could also have a battery-powered arduino inside the box (with SIM breakout) that texted the devops team when opened.

No idea why you were downvoted; it seems like a reasonable idea to me. (Also, IMHO downvoting a good-faith comment like yours is a lazy alternative to posting a substantive response.)

Overcomplex technical solution to a simple problem.

Besides which, if you really want to go full-on with technically clever solutions, keep in mind you could ensure no cellular service prior to opening. But then we're just getting into the realms of silly situations.

Re: SSH Emergency Access

#23

Post by leoh » Sat, Jul 04, 2020, 3:32 PM UTC

isatty wrote: post #15

timeattack wrote: post #14

It's cool and interesting application of the technology, but doesn't really seem to be practical. When you're unable to access machine using your standard SSH keys usually it means that it's highly unlikely that it will be possible to login remotely via other means. As an emergency login there are two common options: * in case of cloud: use remote VM console provided by the hosting provider. * in case of bare-metal:…

Yep, the most common way I've lost access to machines is by messing up the iptables/ipfw rules. Read a post here about avoiding that by having a timed reset with sleep.

Link?

Re: SSH Emergency Access

#24

Post by jon-wood » Sat, Jul 04, 2020, 3:33 PM UTC

marcosdumay wrote: post #21

kevincox wrote: post #16

Earlier quoted context omitted.

I think it depends. I've worked in places that had something like the following setup. - Hardware in datacenters with operators who were not experts on the applications running. - All remote access was done using a short term (~1 day) ssh keys. There was an authentication service to generate these. It was pretty easy to imagine that the authentication service would go down. In this case a selection of people who work…

> All remote access was done using a short term (~1 day) ssh keys. There was an authentication service to generate these. This is weird. Really weird. Did that service use a more secure authentication storage than a password protected key?

It’s really not - by limiting the life of keys, and having a service generating them, you can more effectively lock things down when someone leaves, rather than going round revoking keys from servers. Something we’re experimenting with at work is AWS Instance Connect, which uses your AWS credentials to push a key to a target instance with 1 minute validity - no more managing keys on instances, and revoking access is just a change to an IAM policy.

Re: SSH Emergency Access

#25

Post by oars » Sat, Jul 04, 2020, 3:35 PM UTC

isatty wrote: post #15

timeattack wrote: post #14

It's cool and interesting application of the technology, but doesn't really seem to be practical. When you're unable to access machine using your standard SSH keys usually it means that it's highly unlikely that it will be possible to login remotely via other means. As an emergency login there are two common options: * in case of cloud: use remote VM console provided by the hosting provider. * in case of bare-metal:…

Yep, the most common way I've lost access to machines is by messing up the iptables/ipfw rules. Read a post here about avoiding that by having a timed reset with sleep.

This has happened to me as well. Where could I read about this method?

Re: SSH Emergency Access

#26

Post by lazyant » Sat, Jul 04, 2020, 3:43 PM UTC

isatty wrote: post #15

timeattack wrote: post #14

It's cool and interesting application of the technology, but doesn't really seem to be practical. When you're unable to access machine using your standard SSH keys usually it means that it's highly unlikely that it will be possible to login remotely via other means. As an emergency login there are two common options: * in case of cloud: use remote VM console provided by the hosting provider. * in case of bare-metal:…

Yep, the most common way I've lost access to machines is by messing up the iptables/ipfw rules. Read a post here about avoiding that by having a timed reset with sleep.

For people asking: you can create a resetfw.sh script, for iptables:

  #!/bin/bash

  iptables -P INPUT ACCEPT  
  iptables -P FORWARD ACCEPT  
  iptables -P OUTPUT ACCEPT  
  iptables -t nat -F  
  iptables -t mangle -F  
  iptables -F  
  iptables -X

chmod +x resetfw.sh

and add it for ex to /etc/cron.hourly directory

This way you can test your iptables rules and they'll get clear at every hour. Once you check they are OK you can delete this cronjob.

(NOTE: I'm typing from memory, haven't tested this)

Re: SSH Emergency Access

#27

Post by metiscus » Sat, Jul 04, 2020, 3:51 PM UTC

lazyant wrote: post #26

isatty wrote: post #15

Earlier quoted context omitted.

Yep, the most common way I've lost access to machines is by messing up the iptables/ipfw rules. Read a post here about avoiding that by having a timed reset with sleep.

For people asking: you can create a resetfw.sh script, for iptables: #!/bin/bash iptables -P INPUT ACCEPT iptables -P FORWARD ACCEPT iptables -P OUTPUT ACCEPT iptables -t nat -F iptables -t mangle -F iptables -F iptables -X chmod +x resetfw.sh and add it for ex to /etc/cron.hourly directory This way you can test your iptables rules and they'll get clear at every hour. Once you check they are OK you can delete this cr…

You might add a daily task to remove that task just in case you forget. That way you avoid lockout but don't end up opening yourself up accidentally.

Re: SSH Emergency Access

#28

Post by tyingq » Sat, Jul 04, 2020, 4:11 PM UTC

timeattack wrote: post #14

It's cool and interesting application of the technology, but doesn't really seem to be practical. When you're unable to access machine using your standard SSH keys usually it means that it's highly unlikely that it will be possible to login remotely via other means. As an emergency login there are two common options: * in case of cloud: use remote VM console provided by the hosting provider. * in case of bare-metal:…

A standard for emergency IPMI or other console type type access would be welcome. Vendors have certainly done a bad job in this space. Break-glass type access isn't a new thing.

Re: SSH Emergency Access

#29

Post by tashian » Sat, Jul 04, 2020, 4:12 PM UTC

timeattack wrote: post #14

It's cool and interesting application of the technology, but doesn't really seem to be practical. When you're unable to access machine using your standard SSH keys usually it means that it's highly unlikely that it will be possible to login remotely via other means. As an emergency login there are two common options: * in case of cloud: use remote VM console provided by the hosting provider. * in case of bare-metal:…

Hey there — I'm the author of this post.

There's a few scenarios where I imagined this approach being useful:

* If you have any kind of remote dependency in your SSH auth flow (LDAP, or an online CA, or automated Ansible playbooks to push keys), any of those might fail and render the host otherwise inaccessible.

* It's becoming more common to not ever SSH into machines. So, what if emergency SSH access is the only way to access a host? Some companies even go a few steps further: When a host is SSH'd into, it is considered "tainted by humans", is quarantined and eventually shut down.

* Some hosts should never allow root access to anyone. For example, there's no reason for anyone to have root on a bastion host. So, what if the only way to get root on some hosts is with the emergency key?

While you could use the cloud VM console for emergency access in these cases, having a hardware key provides even more security and would let you turn off cloud VM access.

Of course if you broke your SSHD config, or have a network issue that prevents you from reaching the host, this won't magically fix any of that. IPMI is good for that though.

Re: SSH Emergency Access

#30

Post by tashian » Sat, Jul 04, 2020, 4:17 PM UTC

noodlesUK wrote: post #2

I think another thing we might want to learn about is how to sound the alarm when the break glass is used. Is there an easy way of doing that with SSH? Running a command to page the ops/security team when a server receives a login attempt with an emergency credential?

Hey there, I wrote this post. It's a great question.

One benefit of using certificates for emergency access is that SSHD logging can be configured to show a lot more detail about the certificate that was used. With public keys, there isn't anything to show. But with certificates you have a key ID, serial number, principals, CA fingerprint, etc. So, that log is a good hook for sounding the alarm. A more advanced version of this would allow you to record a reason for using the emergency access key when the connection is made (or when sudo is used).