Earlier quoted context omitted.
I've known SSH certs for a while but never went through the effort of migrating away from keys. I'm very frustrated about manually managing my SSH keys across my different servers and devices though. I assume you gathered a lot of thoughts over these 15 years. Should I invest in making the switch?
A big problem I have with ssh carts is that they are not universally supported. For me, there is always some device or daemon (for example tinyssh in the initramfs of my gaming pc so that I can unlock it remotely) that only works with “plain old ssh keys”. And if I have to distribute and sync my keys onto a few hosts anyway, it takes away the benefits.
SSH certificates: the better SSH experience
71–80 of 137 posts
Re: SSH certificates: the better SSH experience
#72Earlier quoted context omitted.
You will have to manage your SSH CA certificates instead of your keys. The workflows SSH CA's are extremely janky and insecure. With some creative use of `AuthorizedKeysCommand` you can make SSH key rotation painless and secure. With SSH certificates you have to go back to the "keys to the kingdom" antipattern and just hope for the best.
> With SSH certificates you have to go back to the "keys to the kingdom" antipattern and just hope for the best. Whut? This is literally the opposite. With CA certs you can create short-lived certificates, so you can easily grant access to a system for a short time.
Re: SSH certificates: the better SSH experience
#73Earlier quoted context omitted.
> Also, I've never had a security issue due to TOFU, have you? This is a bit like suggesting you've never been in a car crash, so seat belts must not be worth considering. Do you feel that beyond the obvious and documented work in setting them up, there are disadvantages to using SSH certificates?
Certificates provide extra features, like revocation. However, if you do not need the extra features provided by certificates, using SSH-generated keys is strictly equivalent with using certificates and it requires less work. TOFU is neither necessary nor recommended, it is just a convenience feature, to be used when security may be lax. The secure way to use SSH is to never use TOFU but to pair the user and the serv…
That is not the case, and is a major advantage of certificates.
Re: SSH certificates: the better SSH experience
#74Earlier quoted context omitted.
> With SSH certificates you have to go back to the "keys to the kingdom" antipattern and just hope for the best. Whut? This is literally the opposite. With CA certs you can create short-lived certificates, so you can easily grant access to a system for a short time.
And what about the CA?
However, it provides you an additional layer of protection, because it does not need to be on the critical path for every SSH connection. My CA is a Nitrokey HSM, for example. I issue myself temporary certs that are valid only for 6 hours for ephemeral private keys.
Re: SSH certificates: the better SSH experience
#75Earlier quoted context omitted.
The capacity to grant access as a specific remote user is present without certs as well right? The typical authorized_keys file lives under a user directory and grants access only to that user.
The main advantage of certificates is that you are able to do that from the CA without touching the target machine.
You can also make all the certs short-lived (and only store them in ram).
Re: SSH certificates: the better SSH experience
#76Earlier quoted context omitted.
I think the scary reality is most people conflate "keys" and "certificates". I have worked with security engineers that I need to remind that we do not use SSH certs, but rather key auth, and they have to think it through to make it click.
I'm consistently amazed how many developers and security professionals don't have a clear understanding how PPK even works conceptually. Things like deploying dev keys to various production environments, instead of generating/registering them within said environment. One of the worst recent security examples... You can't get this data over HTTPS from $OtherAgency, it's "not secure" ... then their suggestion is a "sec…
Re: SSH certificates: the better SSH experience
#77Every couple of months someone re-discovers SSH certificates, and blogs about them. I'm guilty of it too. My blog post from 15 years ago is nowhere near as good as OP's post, but if I though me of 15 years ago lived up to my standards of today, I'd be really disappointed: https://blog.habets.se/2011/07/OpenSSH-certificates.html
I think the scary reality is most people conflate "keys" and "certificates". I have worked with security engineers that I need to remind that we do not use SSH certs, but rather key auth, and they have to think it through to make it click.
So you get a situation where the lay person is given a "certificate" but it's not really just the certificate it's a PFX file and so e.g. no they mustn't show it you, it has their private key inside it and so you will learn that key and if you're honest you've just ruined their day because they need to start over...
I would say in my career I've had at least two occasions where I did that and I felt awful for the person, because I had set out to help them but now things are worse, and I've had a good number of later occasions where I spent a lot more of their time and mine because I knew I need to be very sure whether their "certificate" is actually a certificate (which they can show me, e.g. Teams message me the file) or a PFX file (thus it has their private key) and I must caution them to show nobody the file yet also try to assist them.
Re: SSH certificates: the better SSH experience
#78With the recent wave of npm hacks stealing private keys, I wanted to limit key's lifetimes. I've set up a couple of yubikeys as SSH CAs on hosts I manage. I use them to create short lived certs (say 24h) at the start of the day. This way i only have to enter the yubikey pin once a day. I could not find an easy way to limit maximum certificate lifetime in openssh, except for using the AuthorizedPrincipalCommand, which…
Instead of using a CA, why not set the key's PIN policy to "once" and use an agent (e.g. https://github.com/FiloSottile/yubikey-agent/ ) that holds an active session to the yubikey? You start the agent at the beginning of the day, enter the PIN once, and then stop the agent at the end of the day.
Re: SSH certificates: the better SSH experience
#79Earlier quoted context omitted.
> TOFU is neither necessary nor recommended Just to make it clear: this does not mean that it is fine to blindly accept the message on first use. The "secure way" implies copying the server's public key as well, which people generally don't do, right? Which is equivalent to verifying the fingerprint shown with the TOFU message, correct?
Like I have said the secure way requires the secure copying of both keys before the first connection attempt The server public key must be copied into "known_hosts" on the client, while the client public key must be copied into "authorized_keys" on the server. When this is the procedure that is always followed, any message shown by SSH about an unknown host means that the connection must be aborted, because the ident…
Sure, I just wanted to make sure that nobody would understand "TOFU is neither necessary nor recommended, just ignore that message and say "yes" when it appears".
Both ends need to be sure of who is on the other end, and there are different ways to achieve that. The way SSH works is that if you haven't copied the server public key locally, it will explicitly ask you to verify it the first time.
I am not sure that "SSH does TOFU". SSH asks you to verify. The human who YOLOs it and approves it without checking is the one doing TOFU, and this is not really secure.
Re: SSH certificates: the better SSH experience
#80SSH certs quietly hurt in prod. Short-lived creds + centralized CA just moves complexity upward without solving the core problem: user management. The system shifts from many small local states to one highly coupled control point. That control point has to be correct and reachable all the time. When it isn’t, failures go wide instead of narrow. Example: a few boxes get popped and start hammering the CA. Now what? Acc…
How do you solve TOFU?
In practice there isn’t really trust on first use: there’s verify the key matches what’s expected, or distribute keys out-of-band (including certs).
If that verification step isn’t happening, then it’s not TOFU, it’s just blind trust.
From an automation/autoscaling angle, the same thing shows up again:
1. either keys are pre-baked / distributed
2. or, something signs them at boot
Signing an instance key is just another way of distributing trust. It doesn’t remove the need for a root of trust, it moves it.
Certificates just add extra steps around the same underlying task.