Security of Infrastructure Secrets
journal.paul.querna.org
Security of Infrastructure Secrets
1–10 of 18 posts
Re: Security of Infrastructure Secrets
#2Storing secrets is fundamentally imperfect ("it's not a secret if someone [or something] else knows it"). This article calls for aid in the form of standards other than PCI-DSS, and those standards do actually exist. NIST 800-53 and 800-130 to name a couple; the EU has others in different industry flavors.
Now, I'm not going to defend these govt. standards as up-to-date or comprehensive. But they're a good philosophical reference for how to manage keys/secrets. Some COTS technologies (which I won't advertise here) try to automate/enforce strong key management for infra, but are typically only affordable for enterprise deployments.
Re: Security of Infrastructure Secrets
#3The attack vectors that surprised me but should not have:
- MongoHQ support person has access to data in customer database.
- CircleCI stores everything in the MongoHQ database, that is used to deploy/control customer servers.
- CircleCI's Customers' CircleCI controlled environments mixed with production environments.
I am guessing everyone just expects most companies, especially those with maybe just Series A financing or close to it, expects those companies to employ this level of security paranoia?
Re: Security of Infrastructure Secrets
#4Needs a [2013] tag. I thought they had been compromised again.
Re: Security of Infrastructure Secrets
#5The link to the MongoHQ page about their compromise is giving a cert expired error (1 day ago), which then redirects to a page not found on compose.io. Odd.
Re: Security of Infrastructure Secrets
#6Disclaimer: infrastructure secrets management is my profession.
This is a lot harder problem than people realize.
If you have a fixed set of machines that need secrets, then encrypting a bag of secrets with each machine's private key works ok.
But in auto scaling / automated / ephemeral scenarios, it doesn't work. You need an RBAC scheme for machines that builds layers of trust; each machine is placed into a role by a trusted service, script or person. Communication between the machines and the secrets service is verified TLS. Each event of access to, or modification of, a secret is recorded for audit purposes. And people and machines should both be treated as first-class actors.
Furthermore, secrets should be kept off permanent media; per the 12factor guidelines, secrets should come from environment variables.
Don't entangle secrets management with other tools like configuration management; otherwise you impede yourself from switching architectures down the road.
Don't create workflows that only ops can control, leaving developers out in the cold, or you are increasing organizational friction.
And if your secrets management processes are opaque to security and compliance people, then they won't have the same level of trust that they would have in a transparent system.
Here's an example of how we approach the problem: http://blog.conjur.net/chef-cookbook-uploads-with-conjur
Re: Security of Infrastructure Secrets
#7Disclaimer: infrastructure secrets management is my profession. This is a lot harder problem than people realize. If you have a fixed set of machines that need secrets, then encrypting a bag of secrets with each machine's private key works ok. But in auto scaling / automated / ephemeral scenarios, it doesn't work. You need an RBAC scheme for machines that builds layers of trust; each machine is placed into a role by…
Something that bothers me is (seemingly) widespread use of passphraseless ssh keys or, using ssh-agent without a timeout setting (so your keys are always loaded). I have to wonder if part of this is because ssh-agent -t starts the timeout clock for automatically unloading keys from whenever the agent was launched, rather than resetting the clock at each signing operation (which would mimic the familiar sudo behaviour).
This makes using ssh-agent with a reasonable timeout incredibly painful.
So you're left with either reentering your passphrase every 5/10/15mins, or basically never. Using smartcards for humans and TPMs for servers is a step in the right direction, but it seems ssh-agent is still missing this basic functionality - or am I missing something?
Re: Security of Infrastructure Secrets
#8Disclaimer: infrastructure secrets management is my profession. This is a lot harder problem than people realize. If you have a fixed set of machines that need secrets, then encrypting a bag of secrets with each machine's private key works ok. But in auto scaling / automated / ephemeral scenarios, it doesn't work. You need an RBAC scheme for machines that builds layers of trust; each machine is placed into a role by…
Something that bothers me is (seemingly) widespread use of passphraseless ssh keys or, using ssh-agent without a timeout setting (so your keys are always loaded). I have to wonder if part of this is because ssh-agent -t starts the timeout clock for automatically unloading keys from whenever the agent was launched, rather than resetting the clock at each signing operation (which would mimic the familiar sudo behaviour…
Most people we have worked with don't take ssh key passwords seriously, because they can be stripped out. We advocated for the idea that password-protected ssh keys are a form of 2-factor auth, but nobody bought into that.
Organizations that want 2-factor auth are typically setting up bastion / jump hosts that require a second factor like a phone-delivered one-time password. This can be configured through the PAM stack.
Once on the bastion, the user can get to other machines within the accessible network using their passwordless ssh key. In effect, each bastion serves as a mini-perimeter.
And yes, people spend a lot of time entering their second factor. Dozens of times per day is not unusual.
Re-reading your question, I'm not really answering it. But maybe this anecdote is useful in some way :-)
Re: Security of Infrastructure Secrets
#9Storing secrets is fundamentally imperfect ("it's not a secret if someone [or something] else knows it"). This article calls for aid in the form of standards other than PCI-DSS, and those standards do actually exist. NIST 800-53 and 800-130 to name a couple; the EU has others in different industry flavors. Now, I'm not going to defend these govt. standards as up-to-date or comprehensive. But they're a good philosophi…
If secrets are rotated or time-limited, they become a lot better. For an example, see the AWS notion of Token Vending Machine.
It's much easier to feel comfortable handing out secrets of each of them had a fixed lifespan. It reduces anxiety greatly.
Re: Security of Infrastructure Secrets
#10Earlier quoted context omitted.
Something that bothers me is (seemingly) widespread use of passphraseless ssh keys or, using ssh-agent without a timeout setting (so your keys are always loaded). I have to wonder if part of this is because ssh-agent -t starts the timeout clock for automatically unloading keys from whenever the agent was launched, rather than resetting the clock at each signing operation (which would mimic the familiar sudo behaviour…
Most people we have worked with don't take ssh key passwords seriously, because they can be stripped out. We advocated for the idea that password-protected ssh keys are a form of 2-factor auth, but nobody bought into that. Organizations that want 2-factor auth are typically setting up bastion / jump hosts that require a second factor like a phone-delivered one-time password. This can be configured through the PAM sta…
> Once on the bastion, the user can get to other machines within the accessible network using their passwordless ssh key.
I really really hope that bastion host never gets compromised.