Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Current Secrets Rotation Process Is Broken

doppler.com

Post reply on HN
Print view View original

11–20 of 22 posts

Re: The Current Secrets Rotation Process Is Broken

#11

Post by NovemberWhiskey »

hbrn wrote: post #4

What's sad is that despite it's current state, secret management has still managed to turn into a cargo cult. It's a "best practice" that people blindly implement without thinking. But secrets are next to useless if they are: - not used to limit number of people that have access to them (it is quite typical in small teams to give everybody access to production, which essentially gives you access to keys) - not regula…

The last part is the key thing. I'm in a continuous running battle with our information security risk team over this - they believe that secret rotation is a secrets management platform feature (and we use Vault so that box is ticked) but the reality is that secret rotation that isn't tied to restarts / reloads of said secrets within distributed application estates is just another way to have an incident.

This is particularly true if you've backed into secrets management as a practice and retrofitted existing platforms; but even if you've designed net new, it's an abstraction-breaker to get reloads of credentials happening all the way down the stack.

Even a blue/green deployment isn't a panacea if there're databases or other shared states that depend on the credential cutover.

From my point of view, the better approach is not to share secrets at all where possible and drive everything from service identity directly. e.g. short-lived credentials like AWS IAM or short-lived PKI proofs like SPIFFE.

Re: The Current Secrets Rotation Process Is Broken

#12

Post by tompic823 »

hbrn wrote: post #4

What's sad is that despite it's current state, secret management has still managed to turn into a cargo cult. It's a "best practice" that people blindly implement without thinking. But secrets are next to useless if they are: - not used to limit number of people that have access to them (it is quite typical in small teams to give everybody access to production, which essentially gives you access to keys) - not regula…

You're exactly right about those two problems, and they actually go together quite well. If a user who previously had access to a secret suddenly has that access revoked, you should rotate that secret. Technically that's only really necessary if the user ever actually saw that secret value, but most secret managers don't expose that information.

At Doppler, we're currently in the process of building out our rotation engine[0]. It allows you to automatically rotate secrets with third party providers (including your database) on a schedule you define. We also provide access logs to see exactly who saw what and when. The ultimate goal is to automatically rotate your secrets as users' access changes. And if you get notification of a breach, you can rotate everything with a single click.

[0] https://docs.doppler.com/docs/secrets-rotation

Re: The Current Secrets Rotation Process Is Broken

#13

Post by nine_k »

jameshart wrote: post #10
hbrn wrote: post #4

What's sad is that despite it's current state, secret management has still managed to turn into a cargo cult. It's a "best practice" that people blindly implement without thinking. But secrets are next to useless if they are: - not used to limit number of people that have access to them (it is quite typical in small teams to give everybody access to production, which essentially gives you access to keys) - not regula…

So much practice around secrets and identity is oriented around assuming secrets are expensive . Like, people will set up a database and create ‘the database user’, with its one password which now needs to be used by all the systems that access that database. To maximize security you need to treat secrets as being cheap . In fact they need to be disposable . In theory any time a system needs access to that database y…

I agree with the idea of cheap, disposable, fine-grained secrets.

With databases in particular the problem is that a connection is a relatively expensive thing which takes noticeable time to establish. Hence connection pooling, and the need for all sessions of a particular class to share credentials.

Re: The Current Secrets Rotation Process Is Broken

#15

Post by feoren »

jameshart wrote: post #10
hbrn wrote: post #4

What's sad is that despite it's current state, secret management has still managed to turn into a cargo cult. It's a "best practice" that people blindly implement without thinking. But secrets are next to useless if they are: - not used to limit number of people that have access to them (it is quite typical in small teams to give everybody access to production, which essentially gives you access to keys) - not regula…

So much practice around secrets and identity is oriented around assuming secrets are expensive . Like, people will set up a database and create ‘the database user’, with its one password which now needs to be used by all the systems that access that database. To maximize security you need to treat secrets as being cheap . In fact they need to be disposable . In theory any time a system needs access to that database y…

> In theory any time a system needs access to that database you could have a process that creates an entirely new user just for that session, with a random password

But doesn't that process just become the new login method? Doesn't that process need its own secrets to have the (much greater) authority to make new users? Doesn't compromising this process give you much more power than compromising one system would have?

Re: The Current Secrets Rotation Process Is Broken

#16

Post by jameshart »

feoren wrote: post #15
jameshart wrote: post #10

Earlier quoted context omitted.

So much practice around secrets and identity is oriented around assuming secrets are expensive . Like, people will set up a database and create ‘the database user’, with its one password which now needs to be used by all the systems that access that database. To maximize security you need to treat secrets as being cheap . In fact they need to be disposable . In theory any time a system needs access to that database y…

> In theory any time a system needs access to that database you could have a process that creates an entirely new user just for that session, with a random password But doesn't that process just become the new login method? Doesn't that process need its own secrets to have the (much greater) authority to make new users? Doesn't compromising this process give you much more power than compromising one system would have…

Right, secret management schemes do have a tendency to feel like ‘but wait, who holds the keys to get hold of the key?’ - it has to be turtles all the way down, right?

Except it turns out not. Secret issuance schemes like this let you move trust boundaries around, they don’t absolutely require all powerful admin systems to exist, and they also let you layer secret schemes on top of more robust trust relationship systems like IAM.

One common pattern is that higher trust ‘control plane’ systems create ephemeral lower trust process instances (eg for autoscaling) - and so patterns like this allow those control planes to bootstrap the child systems with their own unique, cheap, minimally permissioned credentials.

Yes the control plane is a juicier higher priv target. But that’s already the case.

Re: The Current Secrets Rotation Process Is Broken

#17

Post by jiggawatts »

The "better way" for me has been managed identities in Azure. These are Azure Active Directory service principals, which are similar to Group-Managed Service Accounts (gMSA) in Active Directory. They're essentially a platform-managed identity with automatically rotating secrets that the developers never need to know, and sysops don't have to deal with. They can be assigned to VMs or various PaaS services, and then service-to-service access is via standard Azure RBAC, with auditing and everything.

There is nothing to rotate, nothing to put into a "key vault", nothing to worry about leaking out, nothing that could accidentally get pasted into Git, and nothing on that developers' laptop that he left at the airport. You literally say: "I want service A to have read access to service B", and you're done.

Visual Studio automatically injects the developer's personal identity into processes, so that they can access cloud resources during local development, so that's covered too. (For Linux developers, the Azure Identity SDK can pick up the user credentials via the "az" cli.)

"Making fire with stones to avoid a proprietary lighter." -- /user?id=pjmlp

Re: The Current Secrets Rotation Process Is Broken

#18

Post by AlotOfReading »

jchw wrote: post #7
hbrn wrote: post #4

What's sad is that despite it's current state, secret management has still managed to turn into a cargo cult. It's a "best practice" that people blindly implement without thinking. But secrets are next to useless if they are: - not used to limit number of people that have access to them (it is quite typical in small teams to give everybody access to production, which essentially gives you access to keys) - not regula…

I find it frustrating that people still write programs without thinking about secrets rotation. It's kind of understandable that older stuff didn't think about it, since there was certainly a time when people treated secrets, especially things like API keys, as effectively indefinite, but that just doesn't make sense, especially not now. Same for any ephemeral keymatter, or other cryptographic keymatter: there should…

I get what you're saying, but let's not downplay how difficult secret rotation is out the the real world of physical devices.

I work on robots. Let's say you want to rotate the very important image signing keys. If you're lucky enough to have hardware that supports multiple keys (not universal), your process might only be as painful as producing doubly-signed images, coordinating global updates, and field servicing an inevitable percentage of the fleet that had weird issues.

Worst case, you'd have to find every unit around the world and physically replace the PCBs.

I'm personally going to avoid both without some extremely good reasons.

Re: The Current Secrets Rotation Process Is Broken

#19

Post by jchw »

AlotOfReading wrote: post #18
jchw wrote: post #7

Earlier quoted context omitted.

I find it frustrating that people still write programs without thinking about secrets rotation. It's kind of understandable that older stuff didn't think about it, since there was certainly a time when people treated secrets, especially things like API keys, as effectively indefinite, but that just doesn't make sense, especially not now. Same for any ephemeral keymatter, or other cryptographic keymatter: there should…

I get what you're saying, but let's not downplay how difficult secret rotation is out the the real world of physical devices. I work on robots. Let's say you want to rotate the very important image signing keys. If you're lucky enough to have hardware that supports multiple keys (not universal), your process might only be as painful as producing doubly-signed images, coordinating global updates, and field servicing a…

Yep, physical devices are certainly a worst case scenario depending on the constraints. I'm purely talking about software running on servers or, at worst, at least end user computers.

Re: The Current Secrets Rotation Process Is Broken

#20

Post by fvdessen »

jiggawatts wrote: post #17

The "better way" for me has been managed identities in Azure. These are Azure Active Directory service principals, which are similar to Group-Managed Service Accounts (gMSA) in Active Directory. They're essentially a platform-managed identity with automatically rotating secrets that the developers never need to know, and sysops don't have to deal with. They can be assigned to VMs or various PaaS services, and then se…

I'm having the same success with Google IAM, and I agree it completely solves the problem, except for one thing; It only works with compatible services, within your company boundary, within the same cloud provider whereas API keys or oauth flows work everywhere.
Post reply on HN