You should check out Daniel Somerfield's talk at OWASP, "Turtles All the Way Down: Storing Secrets in the Cloud and the Data Center" https://youtu.be/OUSvv2maMYI
Fantastic. Thank you!
Ask HN: In a microservice architecture, how do you handle managing secrets?
31–40 of 60 posts
Re: Ask HN: In a microservice architecture, how do you handle managing secrets?
#32I wrote some of my thoughts on the topic, and the primary motivation behind SOPS [1] (which uses PGP and KMS): https://jve.linuxwall.info/blog/index.php?post/2015/10/01/In...
The initial trust problem boils down to trusting the API that controls the provisioning of your infrastructure. Failing that, you have to ask a human to manually authorize new nodes to retrieve secrets (that's how puppet approves new agent certs).
Re: Ask HN: In a microservice architecture, how do you handle managing secrets?
#33A simple solution if you are in AWS is S3 with instance profiles for access.
This is the solution I've come to use as well. Role-based access in AWS makes a lot of these type of things really nice. Too bad it's not enabled for everything. For example, their hosted ElasticSearch service doesn't yet work with VPC's, and using the role-based access is tough (though possible).
On the subject, I typically store a file containing env variable export statements on S3. When the box is provisioned, the file is downloaded to it. Since the box has role-based access, there is no point in downloading and deleting the file: any process on the box can download it again from S3 at any time. Basically, I trust that the EC2 instance will remain secure. Then the file is source'd in any context where my application code will run.
For applications outside of AWS, I just keep a local non-version-controlled copy of the secrets, and then upload them to the server when I provision it.
Re: Ask HN: In a microservice architecture, how do you handle managing secrets?
#34It also depends on how secret it needs to be. For most of our secrets (those used for configuration) we use Consul.
Do you take advantage of Consul's ACL system them for limiting access to secrets? Also, do you have any form of auditing then when using consul? Thanks for your input!
Can't speak for the parent poster, but over here, yes, we use Consul's ACL. It's pretty solid and easy to use, and the GUI helps a whole lot. In terms of auditing, I've not dug too deeply into that, but there is really good logging.
Re: Ask HN: In a microservice architecture, how do you handle managing secrets?
#35We use Kubernetes, which includes its own secrets API: http://kubernetes.io/v1.1/docs/user-guide/secrets.html I can't remember which issue this was on, but it seemed like there was some discussion on their GitHub project about making pluggable secrets backends (HashiCorp's Vault was mentioned). Kubernetes' secrets API is still very basic, but I think the fundamental concept is very sound and has a great foundation to…
I want to chime in with my experience.
Our service is built on top of docker deployed on CoreOS with fleet and etcd. Most of our secrets & runtime configuration was stored in etcd, which was our attempt to store the config in the environment (http://12factor.net/config).
With Kubernetes, life is much simpler. Gone are the silly fleet configuration files, and the bootstrap scripts I used to configure etcd. Moreover, Kubernetes' secrets volume means I can have my configuration and secrets easily plugged in.
There are definitely other great solutions out there, but I'm sold on Kubernetes.
Re: Ask HN: In a microservice architecture, how do you handle managing secrets?
#36What is the tradeoff matrix for these kinds of services? They all seem pretty similar to me.
Re: Ask HN: In a microservice architecture, how do you handle managing secrets?
#37It also depends on how secret it needs to be. For most of our secrets (those used for configuration) we use Consul.
I think this can be sane when you don't have multiple privilege levels anywhere in the data center you're deploying in. It's less sane if you have less- and more- privileged machines anywhere in the environment, or less- and more- privileged applications.
You're putting a lot of faith in a very complex and not- well- tested codebase if you rely on Consul ACLs to protect secrets.
Re: Ask HN: In a microservice architecture, how do you handle managing secrets?
#38We have an open source solution called Cryptex[0] to handle this. It's better explained by this blog post[1] that gives the thinking and configuration necessary for most scenarios.
[0]: https://github.com/TechnologyAdvice/Cryptex [1]: http://technologyadvice.github.io/lock-up-your-customer-acco...
Re: Ask HN: In a microservice architecture, how do you handle managing secrets?
#39Conjur (https://www.conjur.net/) has been working well the users of our PaaS. It's a self hosted commercial product.
Re: Ask HN: In a microservice architecture, how do you handle managing secrets?
#40We have started work on exposing Hashicorp Vault secrets via FUSE and Docker volumes. The expectation is your containers will just mount secrets via a mount like /secret in the container.
The project is brand new and we'd love to hear your feedback: https://github.com/asteris-llc/vaultfs