Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Environment Variables Considered Harmful for Your Secrets

movingfast.io

Post reply on HN
Print view View original

21–30 of 120 posts

Re: Environment Variables Considered Harmful for Your Secrets

#21

Post by SwellJoe »

aikah wrote: post #10
SwellJoe wrote: post #3

I've always been uncomfortable with the "store config in the environment" part of the 12 factor app thing, since it does imply storing things like database passwords and such, and the argument is that those shouldn't be in files. But, filesystem permissions are reasonably flexible and are easy to reason about (unlike the potential visibility of ENV). I also don't really buy the arguments for ENV storage of even non-s…

> filesystem permissions are reasonably flexible Env makes things really flexible.For instance,you can call a program with env variables directly thus overriding the default ones,which simplify configuring applications. You dont have to have a test set-up,a production-setup or a staging set-up,just start a server or an app with different env variables in the command line. You don't want your config or keys to depend…

Maybe I'm old, but I remember way too many ENV exposure bugs over the years to be particularly confident dumping sensitive information into the ENV. Admittedly, all software should be sanitizing the ENV for untrusted users, but there's just such a long history of people making mistakes. I'm one of those people, in fact. Sure, frameworks and more modern execution models make it easier to avoid those mistakes today, but I've never accidentally checked in a config file with passwords (except a couple times in my t/ test directory, and that only for test server instances). I've made real data exposing security mistakes with ENV.

Re: Environment Variables Considered Harmful for Your Secrets

#22

Post by ntucker »

My company has an internal bit of infrastructure that I think is a somewhat novel approach that allows us to never have any secrets stored unencrypted on disk. There's a server (a set of servers, actually, for redundancy) called the secret server, and its only job is to run a daemon that owns all the secrets. When an app on another server is started up, it must be done from a shell (we use cap) which has an SSH agent forwarded to it. In order for the app to get its database passwords and various other secrets, it makes a request to the secret server (over a TLS-encrypted socket), which checks your SSH identity against an ACL (different identities can have access to different secrets) and does a signing challenge to verify the identity, and if all passes muster, it hands the secrets back. The app process keeps the secrets in memory and your cap shell disconnects, leaving the app unable to fetch any more secrets on your behalf.

The other kink is that the secret server itself reads the secrets from a symmetrically-encrypted file and when it boots, it doesn't actually know how to decrypt it. There's a master key for this that's stored GPG encrypted so that a small number of people can retrieve it and use a client tool that sends the secret server an "unlock" command containing the master key. So any time a secret server reboots, someone with access needs to gpg --decrypt mastersecret | secret_server_unlock_command someserver

There are some obvious drawbacks to this whole system (constraining pushes to require an SSH agent connection is a biggie and wouldn't fly some places, and agent forwarding is not without its security implications) and some obvious problems it doesn't solve (secrets are obviously still in RAM), but on the whole it works very well for distributing secrets to a large number of apps, and we have written tools that have basically completely eliminated any individual's need to ever actually lay eyes on a secret (e.g. if you want to run any tool in the mysql family, there's a tool that fetches the secret for you and spawns the tool you want with MYSQL_PWD temporarily set in the env, so you need not copy/paste it or be tempted to stick it in a .my.cnf).

Re: Environment Variables Considered Harmful for Your Secrets

#23

Post by jph »

We've had good success with distributing our secrets using a GPG-encrypted file that we put in /etc, not in the source code tree. We then use an ENV setting to point the app to the file. This gives us good flexibility (because one server can have multiple GPG files if we want, such as alpha/beta/gamma) and good encryption.

Re: Environment Variables Considered Harmful for Your Secrets

#24

Post by ealexhudson »

mreinsch wrote: post #8
ealexhudson wrote: post #2

Fundamentally, any secrets you store will have some mode of access - there's a downside to each and every way of distributing them. If you're shelling out to commands you think might snarf credentials, the environment is easy for them to pick it out of, but if they're running as the same user then they could probably read the secrets from the config file. If they aren't running as the same user, you need a way of pas…

You're right that a tool which runs under the same user could read your config file and thus could access to your secrets. But there is one main difference: that tool would need to do so explicitly, with the intent of reading (and possibly exposing) your secrets. For me, that's a huge difference from having the secrets being implicitly available to the process through the processes environment.

It doesn't need to do so explicitly: it just needs to have a bug that can encourage it to do so. Most environment leakage also needs to be triggered too.

Re: Environment Variables Considered Harmful for Your Secrets

#25

Post by Someone »

ealexhudson wrote: post #2

Fundamentally, any secrets you store will have some mode of access - there's a downside to each and every way of distributing them. If you're shelling out to commands you think might snarf credentials, the environment is easy for them to pick it out of, but if they're running as the same user then they could probably read the secrets from the config file. If they aren't running as the same user, you need a way of pas…

You could use a kernel service that checksums the calling DLL or even all code loaded in the calling process and compares that against a list of trusted callers before doling out the secret.

Disadvantage, of course, is that you will have to update the list of trusted callers whenever they change. Mac OS X automates that by automatically trusting binaries signed with the same key (trades some security for convenience)

Another disadvantage is that this doesn't work well with scripts (the kernel service would have to know how to find the running scripts in order to checksum them, for every possible scripting language on the system). Also, any form of extension support in a trusted application is problematic.

Re: Environment Variables Considered Harmful for Your Secrets

#27

Post by geocar »

hueving wrote: post #20
aikah wrote: post #10

Earlier quoted context omitted.

> filesystem permissions are reasonably flexible Env makes things really flexible.For instance,you can call a program with env variables directly thus overriding the default ones,which simplify configuring applications. You dont have to have a test set-up,a production-setup or a staging set-up,just start a server or an app with different env variables in the command line. You don't want your config or keys to depend…

>just start a server or an app with different env variables in the command line. Just reference a different config file on the command line. >You don't want your config or keys to depend on an OS,or a language. A path to a config file isn't an OS or language. The ini format is pretty widely supported. >Finally Env variables can be restricted to a set of users,so third party process started with a different one cant a…

    BEGIN {
        $API = new Backend($ENV{credentials});
        delete $ENV{credentials};
    };
Filesystem permissions do not make it possible for a program to internally partition access to those credentials unless you (a) start it as root, or (b) delete the credentials file after reading it.

Re: Environment Variables Considered Harmful for Your Secrets

#28

Post by geocar »

mreinsch wrote: post #17
aikah wrote: post #10

Earlier quoted context omitted.

> filesystem permissions are reasonably flexible Env makes things really flexible.For instance,you can call a program with env variables directly thus overriding the default ones,which simplify configuring applications. You dont have to have a test set-up,a production-setup or a staging set-up,just start a server or an app with different env variables in the command line. You don't want your config or keys to depend…

I agree that ENV variables are useful for general configuration, that's exactly what they were invented for... ENV variables are not restricted by user though, your process can spawn another process under a different user and give it the same environment. It's the nature of the environment that it is usually inherited from the parent which causes the issues when we're talking about secrets.

Delete sensitive environment variables after you read them, or don't run programs you don't trust with an unsanitary environment/argument list (e.g. execve not system)

Re: Environment Variables Considered Harmful for Your Secrets

#29

Post by sritrisna »

Ansible has a neat feature, called Ansible Vault, which lets you encrypt sensitive files. This in combination with dotenv-deployment works pretty well for our Rails Apps. The only thing I’m worried about is someone gaining unauthorised access to our serves and thus being able to read all the credentials stored inside the .env file especially the username & password to our externally hosted db. Probably the only way to prevent this would be “to properly secure your server” and the use of an IDS? Anyone has any experience with someone hacking their servers and successfully preventing e.g. a db dump? In this particular case, how easy would it be to stop attackers in their tracks?

Re: Environment Variables Considered Harmful for Your Secrets

#30

Post by mreinsch »

geocar wrote: post #28
mreinsch wrote: post #17

Earlier quoted context omitted.

I agree that ENV variables are useful for general configuration, that's exactly what they were invented for... ENV variables are not restricted by user though, your process can spawn another process under a different user and give it the same environment. It's the nature of the environment that it is usually inherited from the parent which causes the issues when we're talking about secrets.

Delete sensitive environment variables after you read them, or don't run programs you don't trust with an unsanitary environment/argument list (e.g. execve not system)

Thanks for that idea of deleting sensitive environment variables. I like that for hosters such as heroku which use ENV variables for config (including secrets) by default.
Post reply on HN