The Current Secrets Rotation Process Is Broken
1–10 of 22 posts
Re: The Current Secrets Rotation Process Is Broken
#2The fact that we don't have a well adopted standard for password management, critically including password rotation and key sharing, by now is both surprising and infuriating.
Re: The Current Secrets Rotation Process Is Broken
#3Doppler looks interesting - are there other companies in this space doing good work?
Re: The Current Secrets Rotation Process Is Broken
#4What's sad is that despite it's current state, secret management has still managed to turn into a cargo cult. It's a "best practice" that people blindly implement without thinking.
But secrets are next to useless if they are:
- not used to limit number of people that have access to them (it is quite typical in small teams to give everybody access to production, which essentially gives you access to keys)
- not regularly rotated (at the very least when a person that had access to them leaves the company)
And rotation is hard: a lot of systems still don't support multiple keys, so rotation has to be very carefully tied to some form of blue-green deployment, which is often not possible.
Re: The Current Secrets Rotation Process Is Broken
#5What's sad is that despite it's current state, secret management has still managed to turn into a cargo cult. It's a "best practice" that people blindly implement without thinking. But secrets are next to useless if they are: - not used to limit number of people that have access to them (it is quite typical in small teams to give everybody access to production, which essentially gives you access to keys) - not regula…
>And rotation is hard: a lot of systems still don't support multiple keys, so rotation has to be very carefully tied to some form of blue-green deployment
Oh just today I found out the previous team stored all secrets in git in multiple repositories, and now I need to rotate them all. They encrypted them eventually with sops but did not rotate them so you can find everything in git history. The only solution I can come up with so far is create new credentials and gradually migrate everyone to new secrets, and then delete the old ones. And there's still a chance we can forget something and something will break. I wish there was a button "rotate secrets"...
Re: The Current Secrets Rotation Process Is Broken
#6Are there any widely available, published threat models for secret management of this sort? It seems to me that if you want to frame the discussion in a useful way, you'd start there. Then you can point out where current practice works, where it doesn't, and how your solution works better.
Re: The Current Secrets Rotation Process Is Broken
#7What's sad is that despite it's current state, secret management has still managed to turn into a cargo cult. It's a "best practice" that people blindly implement without thinking. But secrets are next to useless if they are: - not used to limit number of people that have access to them (it is quite typical in small teams to give everybody access to production, which essentially gives you access to keys) - not regula…
I find it frustrating that people still write programs without thinking about secrets rotation. It's kind of understandable that older stuff didn't think about it, since there was certainly a time when people treated secrets, especially things like API keys, as effectively indefinite, but that just doesn't make sense, especially not now. Same for any ephemeral keymatter, or other cryptographic keymatter: there should be a way to have multiple, with at least one "primary" secret and at least one secondary secret. The way I often do this is as simple as having an environment variable/flag/whatever for two separate values and try them in order. Rotating secrets used to salt or encrypt database entries is definitely a bigger pain in the ass, but if it's a big pain in the ass to design a system for it when you're building out a feature, you'd better bet it will be a pain in the ass when you've got a huge security breach and are faced with the task to do it not only in short order, but in concert with many other tasks related to the ongoing fire.
There are definitely other things to be said about the state of secret tokens and secret management, but this one is honestly not that hard to do significantly better at. Rotating secrets should not involve downtime, it should be something you can do out of precaution...
Re: The Current Secrets Rotation Process Is Broken
#8What's sad is that despite it's current state, secret management has still managed to turn into a cargo cult. It's a "best practice" that people blindly implement without thinking. But secrets are next to useless if they are: - not used to limit number of people that have access to them (it is quite typical in small teams to give everybody access to production, which essentially gives you access to keys) - not regula…
I find it frustrating that people still write programs without thinking about secrets rotation. It's kind of understandable that older stuff didn't think about it, since there was certainly a time when people treated secrets, especially things like API keys, as effectively indefinite, but that just doesn't make sense, especially not now. Same for any ephemeral keymatter, or other cryptographic keymatter: there should…
The other day we were sketching out how to integrate another team’s service into ours. They wanted us to compile their API key into our application (in CI/at build time).
Re: The Current Secrets Rotation Process Is Broken
#9Earlier quoted context omitted.
I find it frustrating that people still write programs without thinking about secrets rotation. It's kind of understandable that older stuff didn't think about it, since there was certainly a time when people treated secrets, especially things like API keys, as effectively indefinite, but that just doesn't make sense, especially not now. Same for any ephemeral keymatter, or other cryptographic keymatter: there should…
The other day we were sketching out how to integrate another team’s service into ours. They wanted us to compile their API key into our application (in CI/at build time).
Believe it or not, I've absolutely been there. It blows my mind.
At LEAST let me set the key at runtime. When it comes to stuff that I ship to end users, I'd prefer to be able to dynamically grab the API keys from an endpoint I control for various reasons anyways...
Re: The Current Secrets Rotation Process Is Broken
#10What's sad is that despite it's current state, secret management has still managed to turn into a cargo cult. It's a "best practice" that people blindly implement without thinking. But secrets are next to useless if they are: - not used to limit number of people that have access to them (it is quite typical in small teams to give everybody access to production, which essentially gives you access to keys) - not regula…
So much practice around secrets and identity is oriented around assuming secrets are expensive. Like, people will set up a database and create ‘the database user’, with its one password which now needs to be used by all the systems that access that database.
To maximize security you need to treat secrets as being cheap. In fact they need to be disposable.
In theory any time a system needs access to that database you could have a process that creates an entirely new user just for that session, with a random password - and, bonus, only the permissions that that particular client needs.
If that secret gets compromised, no biggie - you can destroy that user account and the credential will never work again.
Secret rotation is a related approach, but again - if your rotation frequency is ‘quarterly’ rather than ‘hourly’, is there a good reason for that, or have you again just assumed secrets are expensive?