Ask HN: In a microservice architecture, how do you handle managing secrets?

#1

Post by kkamperschroer » Mon, Jan 18, 2016, 9:20 PM UTC

I'm evaluating solutions for secrets management in relation to a distributed microservice architecture and am curious to hear what everyone else out there does. Some options I've considered:

- Git-crypt and deploying secrets along with binaries

- Hashicorp Vault

- Square Keywhiz

- AWS KMS

- Lyft Confidant

- Roll your own

All seem to have pros and cons depending on use cases and how mission critical the service you are offering is.

So what do you do to solve this problem in your world?

Re: Ask HN: In a microservice architecture, how do you handle managing secrets?

#2

Post by PaulHoule » Mon, Jan 18, 2016, 9:25 PM UTC

When I am building stuff in AWS, most of the secrets are for entities that are access-controlled by AWS and can be passed in through server roles.

Re: Ask HN: In a microservice architecture, how do you handle managing secrets?

#3

Post by gtaylor » Mon, Jan 18, 2016, 9:40 PM UTC

We use Kubernetes, which includes its own secrets API:

http://kubernetes.io/v1.1/docs/user-guide/secrets.html

I can't remember which issue this was on, but it seemed like there was some discussion on their GitHub project about making pluggable secrets backends (HashiCorp's Vault was mentioned).

Kubernetes' secrets API is still very basic, but I think the fundamental concept is very sound and has a great foundation to continue building on.

Re: Ask HN: In a microservice architecture, how do you handle managing secrets?

#4

Post by late2part » Mon, Jan 18, 2016, 9:42 PM UTC

I'm pretty happy with this solution from Strongauth.

http://keyappliance.strongauth.com/

You can secure the root for it with TPM or HSM.

Re: Ask HN: In a microservice architecture, how do you handle managing secrets?

#5

Post by alpb » Mon, Jan 18, 2016, 9:43 PM UTC

gtaylor wrote: post #3

We use Kubernetes, which includes its own secrets API: http://kubernetes.io/v1.1/docs/user-guide/secrets.html I can't remember which issue this was on, but it seemed like there was some discussion on their GitHub project about making pluggable secrets backends (HashiCorp's Vault was mentioned). Kubernetes' secrets API is still very basic, but I think the fundamental concept is very sound and has a great foundation to…

Docker's commercial offering DUCP (Docker Universal Control Plane) offers this feature as well. Out in the wild, you can find Docker volume drivers for Keywhiz etc [1] that makes secrets available as files mounted to a container. I think Kubernetes does this, too.

If you are running on cloud, you would probably want your cloud provider to give you service secrets and rotate them somehow. AWS/Google Compute metadata service or Azure Key Vault are capable of doing this but I don't think they entirely map the microservices world because ACLs are set on the VM instances, not microservices specifically.

[1] https://github.com/calavera/docker-volume-keywhiz

Re: Ask HN: In a microservice architecture, how do you handle managing secrets?

#6

Post by Crystalin » Mon, Jan 18, 2016, 9:43 PM UTC

It also depends on how secret it needs to be. For most of our secrets (those used for configuration) we use Consul.

Re: Ask HN: In a microservice architecture, how do you handle managing secrets?

#7

Post by avitzurel » Mon, Jan 18, 2016, 9:46 PM UTC

I'd choose the one that I am the most comfortable with and is less obtrusive to the rest of my stack.

Whatever you choose, make sure you are comfortable with it, it's easy to deploy and work with.

Re: Ask HN: In a microservice architecture, how do you handle managing secrets?

#8

Post by ejp » Mon, Jan 18, 2016, 9:47 PM UTC

Another option for your list, which you'll have to evaluate for your use case: https://wiki.openstack.org/wiki/Barbican

I've been evaluating most of these same options for my use case, but haven't made any decisions yet.

Re: Ask HN: In a microservice architecture, how do you handle managing secrets?

#9

Post by yodon » Mon, Jan 18, 2016, 9:49 PM UTC

it's a huge pain point for us. We're a .NET shop rolling our own that mimics/overlays app.config and web.config patterns for both dev and production usage. Our concern is less on how do you get the secrets to the box (though that's obviously important) and more on how do you keep an attacker who has started penetrating your infrastructure from gaining control of the infrastructure that holds your secrets.

Re: Ask HN: In a microservice architecture, how do you handle managing secrets?

#10

Post by kkamperschroer » Mon, Jan 18, 2016, 9:49 PM UTC

avitzurel wrote: post #7

I'd choose the one that I am the most comfortable with and is less obtrusive to the rest of my stack. Whatever you choose, make sure you are comfortable with it, it's easy to deploy and work with.

Thanks for the tip. I guess the easiest thing to do is use git-crypt with some encrypted file and have the secrets available at deploy time, but I'm worried about long term disadvantages to this approach. Rolling secrets would then require a deployment of at least that secrets file and restarting the services, or writing them in a way they read the file every time they need the secret.

Since our stack isn't on AWS, it kind of throws out AWS KMS and Lyft Confidant (since it is built on AWS). I'll keep digging into Vault and the other options put forward in this thread. Thanks again.