Crypto Tools for DevOps: Git-Crypt – Tozny
1–10 of 39 posts
Re: Crypto Tools for DevOps: Git-Crypt – Tozny
#2Don't keep encrypted secrets in your git repositories, if for no other reason than that it makes access revocation deceptively difficult --- but also because it encourages you to have a development team in which ordinary devs have a full complement of secrets on their laptops at all times.
Instead, keep secrets "out of band" and supply them to applications as part of your deployment process.
Re: Crypto Tools for DevOps: Git-Crypt – Tozny
#3Don't keep encrypted secrets in your git repositories, if for no other reason than that it makes access revocation deceptively difficult --- but also because it encourages you to have a development team in which ordinary devs have a full complement of secrets on their laptops at all times. Instead, keep secrets "out of band" and supply them to applications as part of your deployment process.
Just curious - what's the preferred way of passing secrets from the deployment process to the application?
Environment variables? Simple to use and understand but can leak to child processes and crash reports.
Files on disk? Usually can also be read by children.
Via stdin on app startup? Seems simple...
Some other way?
Re: Crypto Tools for DevOps: Git-Crypt – Tozny
#4Don't keep encrypted secrets in your git repositories, if for no other reason than that it makes access revocation deceptively difficult --- but also because it encourages you to have a development team in which ordinary devs have a full complement of secrets on their laptops at all times. Instead, keep secrets "out of band" and supply them to applications as part of your deployment process.
Just curious - what's the preferred way of passing secrets from the deployment process to the application? Environment variables? Simple to use and understand but can leak to child processes and crash reports. Files on disk? Usually can also be read by children. Via stdin on app startup? Seems simple... Some other way?
Personally I usually go with env vars for small, one-off deployments. For larger environments with multiple deployables/teams, a tool like Vault (https://www.vaultproject.io/) is pretty handy.
Re: Crypto Tools for DevOps: Git-Crypt – Tozny
#5Don't keep encrypted secrets in your git repositories, if for no other reason than that it makes access revocation deceptively difficult --- but also because it encourages you to have a development team in which ordinary devs have a full complement of secrets on their laptops at all times. Instead, keep secrets "out of band" and supply them to applications as part of your deployment process.
We ran into this problem with terraform and we needed a fix quickly. The problem is that they recommend you check in your .tfstate files, which does make sense; they do need to be synchronized between everyone who might be working on the repo. However, we learned later down the line that in some cases, the state files might contain secrets. So we rolled out git-crypt for all .tfstate files to future-proof ourself against accidental checkin of secrets.
Terraform does support a number of out of band state management backends that I would prefer to use, but none of those backends support encryption at rest. Hopefully hashicorp will roll out support for vault as a backend at some point...
Re: Crypto Tools for DevOps: Git-Crypt – Tozny
#6Don't keep encrypted secrets in your git repositories, if for no other reason than that it makes access revocation deceptively difficult --- but also because it encourages you to have a development team in which ordinary devs have a full complement of secrets on their laptops at all times. Instead, keep secrets "out of band" and supply them to applications as part of your deployment process.
Just curious - what's the preferred way of passing secrets from the deployment process to the application? Environment variables? Simple to use and understand but can leak to child processes and crash reports. Files on disk? Usually can also be read by children. Via stdin on app startup? Seems simple... Some other way?
What app? They all take credentials different ways. The ideal would be an unprivileged child talking to a privileged parent, where the child asks for the secret and is authorized by the parent, and then both the child and parent erase the [unencrypted] secret when not in use. If your app doesn't do this, executing some app that then feeds the secret via a file descriptor (pipe/socket/etc) (and hopefully erases all traces of it after) is similar.
The simplest hack is to use a configuration management tool to start your app, and have its pre-exec step provide the credential, and post-exec remove any trace of it (which, again, will not work everywhere). The config management tool can be authorized to pull the credential from a remote host by some public-key protocol and a hole in your firewall, or by a local cache of secrets pushed by the config management tool only to the hosts and accounts that specifically need them. These have security and reliability tradeoffs.
You can also use a credential management suite for a more robust solution, of which there are both commercial and free options.
Re: Crypto Tools for DevOps: Git-Crypt – Tozny
#7Don't keep encrypted secrets in your git repositories, if for no other reason than that it makes access revocation deceptively difficult --- but also because it encourages you to have a development team in which ordinary devs have a full complement of secrets on their laptops at all times. Instead, keep secrets "out of band" and supply them to applications as part of your deployment process.
We ran into this problem with terraform and we needed a fix quickly. The problem is that they recommend you check in your .tfstate files, which does make sense; they do need to be synchronized between everyone who might be working on the repo. However, we learned later down the line that in some cases, the state files might contain secrets. So we rolled out git-crypt for all .tfstate files to future-proof ourself aga…
Why not store your remote state in S3 buckets? They support encryption at rest.
Re: Crypto Tools for DevOps: Git-Crypt – Tozny
#8Earlier quoted context omitted.
Just curious - what's the preferred way of passing secrets from the deployment process to the application? Environment variables? Simple to use and understand but can leak to child processes and crash reports. Files on disk? Usually can also be read by children. Via stdin on app startup? Seems simple... Some other way?
What app? They all take credentials different ways. The ideal would be an unprivileged child talking to a privileged parent, where the child asks for the secret and is authorized by the parent, and then both the child and parent erase the [unencrypted] secret when not in use. If your app doesn't do this, executing some app that then feeds the secret via a file descriptor (pipe/socket/etc) (and hopefully erases all tr…
> "The simplest hack is to use a configuration management tool"
> "You can also use a credential management suite for a more robust solution"
You have to start somewhere, this is why automated system with credentials isn't a simple task to solve.
Imagine this, you use consul to store your secrets, as you wrote up a script for Ansible to look things up, this include secrets. Great, but you need to authorize. So you need to keep the token somewhere. It's okay if you have a person typing in, but if you deploy through Jenkins on a regular basis automatically, you need to add that token as a password to Jenkins's data store.
But you provision and configure your Jenkins using your Ansible playbook, and your playbook needs to get to Consul to get to the secret so you can compare the key on Jenkins. Now this is getting tricker.
Okay, say you are on EC2, you can use instance profile to grant instance access to S3 / DynamoDB where you put your ultimate secrets. But if you are paranoid or for some reason you decided to do client-side encryption and keeping the master key to yourself (you generate them and you keep them, never given out to AWS), then you are back to square one. Where the heck do you keep your master key safe? KMS? No? In your own Consul? Okay where should we keep the token? Ugh.
What if you don't use AWS? Google Cloud?
You have to trust at least one place. If your repo can be trusted, let it be. There are three major factors in combating a system that has to put trust in password/cert/key:
* keep rotate the credentials
* everyone should pull from the same credentials management infrastructure
* least privilege
But #2 takes a while to complete, mainly because now you need to develop / rewrite your script on Jenkins to not use Jenkins's password datastore, and the password masking plugin. You have to develop your own masking plugin.
The most trusted place is where you will need to add extreme security measure. Assuming the encryption can't be broken in reasonable amount of time, the only way to keep a secret unrecoverable is if the secret holder is dead ("only one can live if two people know a secret).
Re: Crypto Tools for DevOps: Git-Crypt – Tozny
#9Earlier quoted context omitted.
Just curious - what's the preferred way of passing secrets from the deployment process to the application? Environment variables? Simple to use and understand but can leak to child processes and crash reports. Files on disk? Usually can also be read by children. Via stdin on app startup? Seems simple... Some other way?
What app? They all take credentials different ways. The ideal would be an unprivileged child talking to a privileged parent, where the child asks for the secret and is authorized by the parent, and then both the child and parent erase the [unencrypted] secret when not in use. If your app doesn't do this, executing some app that then feeds the secret via a file descriptor (pipe/socket/etc) (and hopefully erases all tr…
> (and hopefully erases all traces of it after)
Which is a "best effort" at best.
Re: Crypto Tools for DevOps: Git-Crypt – Tozny
#10Earlier quoted context omitted.
What app? They all take credentials different ways. The ideal would be an unprivileged child talking to a privileged parent, where the child asks for the secret and is authorized by the parent, and then both the child and parent erase the [unencrypted] secret when not in use. If your app doesn't do this, executing some app that then feeds the secret via a file descriptor (pipe/socket/etc) (and hopefully erases all tr…
> "The simplest hack is to use a configuration management tool" > "You can also use a credential management suite for a more robust solution" You have to start somewhere, this is why automated system with credentials isn't a simple task to solve. Imagine this, you use consul to store your secrets, as you wrote up a script for Ansible to look things up, this include secrets. Great, but you need to authorize. So you ne…
>> "The simplest hack is to use a configuration management tool"
>> "You can also use a credential management suite for a more robust solution"
> Imagine this, you use consul to store your secrets, as you wrote up a script for Ansible to look things up, this include secrets. Great, but you need to authorize. So you need to keep the token somewhere.
You see, the problem springs from you mistaking Ansible for configuration management tool. It is not, it is somebody's deployment script. What you want is CFEngine, Puppet, or Chef, and you need it download configuration templates and fill them with secrets on the server that you want configured. (CFEngine can easily do that; Chef -- probably, since it's not a standalone tool, but a Ruby framework, so you have ERB at your disposal; Puppet? no idea if it can use ERB in masterless mode.)
Now you need to solve the problem how to get the secrets to the server being configured. This would be easy if you had a specialized database service that ships credentials, and only the ones that are relevant to the server are sent. It's a pity we don't have such a database (or at least I am not aware of any), but it's not exactly a famine problem to solve, it's just juggling with access lists and some simple log replication protocol.