Ask HN: In a microservice architecture, how do you handle managing secrets?

1. create a secret key, store it with the code of your App 2. store the encrypted credentials in env vars

Later on we even launched our own solution for our clients, an app_secrets.yml file, which can be edited via Dashboard. http://help.fortrabbit.com/secrets

The nice thing is, that this file is partly managed by the platform for it's own credentials and partly by the user.

That has been running for a while now. The adaption rate is low until now. It turned out that not everything will fit into that ONE fault. Blackfire.io and NewRelic run as PHP extensions, thus the API-keys are stored with the extension setting.

We have also discussed to implement an some open source "Secret as a Service" but came to the conclusion that this can too easily turn into to be a SPOF.

I am amazed that this topic is getting discussed again and I have learned about many new concepts here.

At startup the container reaches out to the Secretary daemon that holds the master keys, using public key cryptos to authenticate itself. The Secretary deamon uses Marathon to authenticate containers (checking their public keys stored in env vars) and validate that they're authorized for the specific secret in question (checking that the encrypted secret is indeed part of the containers env vars).

Meaning that Marathon is the single source of trust of which container can access what secrets. The problem then becomes controlling who and how changes are made to the Git repo containing the CD config, which is something Github does well with roles, status/deployment API and pull requests.

We had a similar problem as some describe with the distribution of the initial secret (i.e. Vault token) and one time Vault tokens being cumbersome in dynamic scaling envs. We didn't want the cleartext token ending up in config files nor in the https://github.com/meltwater/lighter config we use to drive our continuous delivery pipelines that go into Mesos/Marathon. We also had some other aspects like

* Wanting to keep secrets, app config and code versions promoted together throughout or deployment pipelines. Seeing secrets as another type of app config we wanted to track all config and versions for an app in the same way, in the same place to avoid mismatches or deployment dependencies.

* Wanting to enable our very independent devteams to easily manage secrets for their services, same was as they manage the app config, versions and rollout of their services. And delegate management of what service is authorized for what secrets to devteams (with both automated checks for unencrypted secrets, and some gentle manual coaching post-commit)

* Versioning and rolling upgrades for secrets? E.g. how to roll out a new secret in a Marathon rolling upgrade? Creating and managing versioned keys in Vault seemed somewhat cumbersome.

Perhaps something like that could be used to solve your initial secret distribution problem or even handle the secrets themselves until Vault has solved the initial secret problem..?

Also on access control, any process with root on any node in your cluster can get access to all your secrets (since the kubelet needs to be able to do so). There are no user access controls either; any cluster admin can dump all the secrets.

This stuff is clearly documented, so it's not an indictment on k8s; I just get the feeling that the feature isn't really ready for production use yet.