Live data from Hacker News

SSH certificates: the better SSH experience

jpmens.net

31–40 of 137 posts

Re: SSH certificates: the better SSH experience

#31

Anyone tried out Userify? It creates/removes ssh pubkeys locally so (like a CA) no authn server needs to be online. But unlike certs, active sessions and processes are terminated when the user access is revoked.

We're in the process of updating the experience to this century! ;)

We've always taken the stance that crusty is better than vulnerable, but it turns out that not having a modern experience after 15 years is starting to feel like maybe we need to step up the features and shininess :)

Re: SSH certificates: the better SSH experience

#32
post #23

All those articles about SSH certificates fall short of explaining how the revocation list can/should be published. Is that yet another problem that I need to solve with syncthing? https://man.openbsd.org/ssh-keygen.1#KEY_REVOCATION_LISTS

If you generate short lived certificates via an automated process/service then you don’t really need to manage a revocation list as they will have expired in short order.

But then you can't log in if your box goes offline for any reason.

Re: SSH certificates: the better SSH experience

#33

Earlier quoted context omitted.

If you generate short lived certificates via an automated process/service then you don’t really need to manage a revocation list as they will have expired in short order.

But then you can't log in if your box goes offline for any reason.

Hmm. For user certs you can have the service sign them for, say an hour, so long as you can ssh to your server in that time then there’s no need for any other interaction.

Sure you need your signing service to be reasonably available, but that’s easily accomplished.

Maybe I misunderstand?

Re: SSH certificates: the better SSH experience

#34
post #13

Earlier quoted context omitted.

> Also, I've never had a security issue due to TOFU, have you? This is a bit like suggesting you've never been in a car crash, so seat belts must not be worth considering. Do you feel that beyond the obvious and documented work in setting them up, there are disadvantages to using SSH certificates?

Certificates provide extra features, like revocation. However, if you do not need the extra features provided by certificates, using SSH-generated keys is strictly equivalent with using certificates and it requires less work. TOFU is neither necessary nor recommended, it is just a convenience feature, to be used when security may be lax. The secure way to use SSH is to never use TOFU but to pair the user and the serv…

> TOFU is neither necessary nor recommended

Just to make it clear: this does not mean that it is fine to blindly accept the message on first use.

The "secure way" implies copying the server's public key as well, which people generally don't do, right? Which is equivalent to verifying the fingerprint shown with the TOFU message, correct?

Re: SSH certificates: the better SSH experience

#35
I've had very good experiences with SSH Communication Security company's (the guys who invented SSH) PrivX product to manage secure remote access, including SSH certificates and also cert based Windows authentication. It supports other kinds of remote targets too, via webui or with native clients. Great product.

Re: SSH certificates: the better SSH experience

#36
SSH certs quietly hurt in prod. Short-lived creds + centralized CA just moves complexity upward without solving the core problem: user management.

The system shifts from many small local states to one highly coupled control point. That control point has to be correct and reachable all the time. When it isn’t, failures go wide instead of narrow.

Example: a few boxes get popped and start hammering the CA. Now what? Access is broken everywhere at once.

Common friction points:

     1. your signer that has to be up and correct all the time
     2. trust roots everywhere (and drifting)
     3. TTL tuning nonsense (too short = random lockouts, too long = what was the point)
     4. limited on-box state makes debugging harder than it should be
     5. failures tend to fan out instead of staying contained
Revocation is also kind of a lie. Just waiting for expiry and hoping that’s good enough.

What actually happens is people reintroduce state anyway: sidecars, caches, agents… because you need it.

We went the opposite direction:

     1. nodes pull over outbound HTTPS
     2. local authorized_keys is the source of truth locally
     3. users/roles are visible on the box
     4. drift fixes itself quickly
     5. no inbound ports, no CA signatures (WELL, not strictly true*!)
You still get central control, but operation and failure modes are local instead of "everyone is locked out right now."

That’s basically what we do at Userify (https://userify.com). Less elegant than certs, more survivable at 2am. Also actually handles authz, not just part of authn.

And the part that usually gets hand-waved with SSH CAs:

     1. creating the user account
     2. managing sudo roles
     3. deciding what happens to home directories on removal
     4. cleanup vs retention for compliance/forensics
Those don’t go away - they're just not part of the certificate solution.

* (TLS still exists here, just at the transport layer using the system trust store. That channel delivers users, keys, and roles. The rest is handled explicitly instead of implied.)

Re: SSH certificates: the better SSH experience

#37
post #13

Earlier quoted context omitted.

> Also, I've never had a security issue due to TOFU, have you? This is a bit like suggesting you've never been in a car crash, so seat belts must not be worth considering. Do you feel that beyond the obvious and documented work in setting them up, there are disadvantages to using SSH certificates?

Your ISP or telecom has to be compromised for TOFU to be relevant to anything. In practice that never happens.

Not just your ISP. If an attacker slipped a device onto your LAN and also you happened to be sshing to a new box for the first time then TOFU poses a problem. But that's an awfully limited attack surface. It's similar to the difference between leaking a fax while it's sent versus leaking years old emails that are just sitting there on an internet accessible server.

As for your ISP I think you should never rely on TOFU over the public internet. If you really don't want to do ssh certs it's easy enough to make the host key available securely via https.

Re: SSH certificates: the better SSH experience

#38

Every couple of months someone re-discovers SSH certificates, and blogs about them. I'm guilty of it too. My blog post from 15 years ago is nowhere near as good as OP's post, but if I though me of 15 years ago lived up to my standards of today, I'd be really disappointed: https://blog.habets.se/2011/07/OpenSSH-certificates.html

oh man, I referred back to your blog post when I wrote the ssh certificate authority for $job ... ~10 years ago.

Thank for writing it!

Re: SSH certificates: the better SSH experience

#39
post #3

Every couple of months someone re-discovers SSH certificates, and blogs about them. I'm guilty of it too. My blog post from 15 years ago is nowhere near as good as OP's post, but if I though me of 15 years ago lived up to my standards of today, I'd be really disappointed: https://blog.habets.se/2011/07/OpenSSH-certificates.html

I've known SSH certs for a while but never went through the effort of migrating away from keys. I'm very frustrated about manually managing my SSH keys across my different servers and devices though. I assume you gathered a lot of thoughts over these 15 years. Should I invest in making the switch?

I am keeping an eye on the new (and alpha) Authentik agent which will allow idp based ssh logins. There's also SSSD already supported but it requires glibc (due to needing NSS) meaning it's not available on Alpine.

Re: SSH certificates: the better SSH experience

#40

Earlier quoted context omitted.

But then you can't log in if your box goes offline for any reason.

Hmm. For user certs you can have the service sign them for, say an hour, so long as you can ssh to your server in that time then there’s no need for any other interaction. Sure you need your signing service to be reasonably available, but that’s easily accomplished. Maybe I misunderstand?

That works for authn in the happy path: short-lived cert, grab it, connect, done.

Except for everything around that:

* user lifecycle (create/remove/rename accounts)

* authz (who gets sudo, what groups, per-host differences)

* cleanup (what happens when someone leaves)

* visibility (what state is this box actually in right now?)

SSH certs don’t really touch any of that. They answer can this key log in right now, not what should exist on this machine.

So in practice, something else ends up managing users, groups, sudoers, home dirs, etc. Now there are two systems that both have to be correct.

On the availability point: "reasonably available" is doing a lot of work ;)

Even with 1-hour certs:

* new sessions depend on the signer

* fleet-wide issues hit everything at once

* incident response gets awkward if the signer is part of the blast radius

The failure mode shifts from a few boxes don't work to nobody can get in anywhere

The pull model just leans the other way:

* nodes converge to desired state

* access continues even if control plane hiccups

* authn and authz live together on the box

Both models can work - it’s more about which failure mode is tolerable to you.

Post reply on HN