AWS Secrets Manager – Store, Distribute, and Rotate Credentials Securely • Hacker News Board

Re: AWS Secrets Manager – Store, Distribute, and Rotate Credentials Securely

#121

Post by jessaustin » Thu, Apr 05, 2018, 12:47 PM UTC

tptacek wrote: post #32

toomuchtodo wrote: post #25

Earlier quoted context omitted.

> The only value-add is secret rotation and that's not something that most of the time makes sense to use. From a security perspective, you should be rotating secrets somewhere between annually and every 90 days, depending on your business/security/compliance requirements and the nature of the data secured by the secret.

It depends on the secret and the degree to which the secret is exposed. SSH creds should get rotated constantly ; a one-hour SSH login cred is a significant exposure. But an API secret that is kept in Parameter Store and not exposed to developers doesn't really benefit from rotation every 3 months in proportion to the amount of mechanism required to do that.

...a one-hour SSH login cred is a significant exposure.

Does this refer to an SSH password or e.g. a 4096-bit RSA key, or possibly something completely different? If this refers to an RSA key then I'm quite confused.

Re: AWS Secrets Manager – Store, Distribute, and Rotate Credentials Securely

#122

Post by mlrtime » Thu, Apr 05, 2018, 1:37 PM UTC

niklaslindblad wrote: post #119

scrollaway wrote: post #18

First reaction: Holy crap! They finally turned ParameterStore into a proper product! Second reaction: Holy shit that's expensive [for what it does]. ParameterStore is free (minus the KMS component). The only value-add is secret rotation and that's not something that most of the time makes sense to use. [Edit: I'm not advocating for no rotation; see replies] Edit: Had more time to think about it. Someone enlighten me:…

After Amazon EC2 Systems Manager Parameter Store got some much needed love last year (adding KMS encryption and versioning), I wrote a small utility for populating environment variables using a specific Parameter Store prefix: https://github.com/nlindblad/confidential It can either write the environment variables out to a Docker compatible environment file or it can be used as a wrapper to run any executable/script w…

Aren't all your secrets available as environment variables on your instances? If someone has shell access or gets the environment access pragmatically, all your secrets are compromised.

Re: AWS Secrets Manager – Store, Distribute, and Rotate Credentials Securely

#123

Post by mlrtime » Thu, Apr 05, 2018, 1:41 PM UTC

scrollaway wrote: post #27

toomuchtodo wrote: post #25

Earlier quoted context omitted.

> The only value-add is secret rotation and that's not something that most of the time makes sense to use. From a security perspective, you should be rotating secrets somewhere between annually and every 90 days, depending on your business/security/compliance requirements and the nature of the data secured by the secret.

You're not necessarily your own source of secret (and even when you are, you don't necessarily have the option to use AWS-sourced rotation). In other words, yes, you should be rotating what you can rotate, but this doesn't always help. Furthermore, it doesn't justify the pricing. It would make sense if this were, say, "Hey, you can now auto-rotate SSM-PS secrets for an extra $0.40/secret". Right now it just seems wei…

Does it matter? How many secrets do you have where this is even close to your ec2/storage costs?

Re: AWS Secrets Manager – Store, Distribute, and Rotate Credentials Securely

#124

Post by jeremya » Thu, Apr 05, 2018, 2:15 PM UTC

mattwad wrote: post #78

mratzloff wrote: post #66

Earlier quoted context omitted.

Well, you can use KMS to do something similar, but without the auto-rotation. 1. Use a KMS key to encrypt a secrets file (obviously, never check this into source control) 2. Store the encrypted secrets file in an S3 bucket 3. Tie a new IAM role with kms:Decrypt and s3:GetObject policies for the relevant resources to your EC2 instance 4. On app start, get the KMS key and secrets file, decrypt, and set environment vari…

This seems like the easiest solution, and we've been pretty successful. We have sub-folders based on your environment and only machines have roles that can read production. We also realized you don't even have to encrypt and decrypt.. just turn on encryption for the S3 bucket, I believe it's the same thing.

It's not exactly the same thing. Encryption at rest on S3 is good, but security is about layers. If you only use that, then you are one mis-configured S3 bucket policy away from exposing your secrets. Bad bucket policies are one of AWS' biggest footguns- plenty of intelligent people have done it before, don't assume it's a mistake you'll never make. It's so easy because S3 is a service that has to make publicly-accessible storage easy to achieve.

KMS keys on the other hand, do not have that use case. If the secrets in your S3 bucket have been encrypted using a KMS key (or better yet, a data key derived from a KMS key) and your bucket is compromised, your secrets are safe as long as the attacker cannot ALSO get access to the KMS key. By all means, enable encryption at rest on S3, but don't make that your only defense.

Re: AWS Secrets Manager – Store, Distribute, and Rotate Credentials Securely

#125

Post by gregmac » Thu, Apr 05, 2018, 4:28 PM UTC

toomuchtodo wrote: post #25

scrollaway wrote: post #18

First reaction: Holy crap! They finally turned ParameterStore into a proper product! Second reaction: Holy shit that's expensive [for what it does]. ParameterStore is free (minus the KMS component). The only value-add is secret rotation and that's not something that most of the time makes sense to use. [Edit: I'm not advocating for no rotation; see replies] Edit: Had more time to think about it. Someone enlighten me:…

> The only value-add is secret rotation and that's not something that most of the time makes sense to use. From a security perspective, you should be rotating secrets somewhere between annually and every 90 days, depending on your business/security/compliance requirements and the nature of the data secured by the secret.

What's the rationale behind rotating? I've never really understood this. I'm asking this from a fundamental point-of-view: I realize there are policies and standards that require rotation, but I want to know why those policies say that in the first place.

If you have a secret, it's either (potentially) compromised and should be changed immediately, or it's not compromised and thus doesn't need to be changed.

If you're not sure if it's compromised or not, you have an entirely different problem that isn't solved by rotating. Put another way, rotating every 90 days is the equivalent of saying "it's okay if this secret is compromised, because no one can do any damage with it if they only have under 90 days" -- and I can't imagine any situation in which that would be true or acceptable.

Re: AWS Secrets Manager – Store, Distribute, and Rotate Credentials Securely

#126

Post by AnimalMuppet » Thu, Apr 05, 2018, 4:39 PM UTC

gregmac wrote: post #125

toomuchtodo wrote: post #25

Earlier quoted context omitted.

> The only value-add is secret rotation and that's not something that most of the time makes sense to use. From a security perspective, you should be rotating secrets somewhere between annually and every 90 days, depending on your business/security/compliance requirements and the nature of the data secured by the secret.

What's the rationale behind rotating? I've never really understood this. I'm asking this from a fundamental point-of-view: I realize there are policies and standards that require rotation, but I want to know why those policies say that in the first place. If you have a secret, it's either (potentially) compromised and should be changed immediately, or it's not compromised and thus doesn't need to be changed. If you'r…

You seem to be thinking of a detected compromise, which may have compromised secret X, and your response is to change secret X right now. And I think that's the correct response.

But in another sense, all your secrets are potentially compromised, all the time. That is, there are compromises that you don't detect, or don't detect right away. They happen. What should you do? Well, you change the secrets every 90 days, even if you don't (yet) know that they have been compromised, to minimize the damage of a compromise that you don't know about.

But, you may say, by that logic shouldn't we change them every day? Every hour? But the other tradeoff is with annoying your users, wasting peoples' time on changing secrets, wasting time as they have to try to remember today's secret, and all the associated hassle. It's a compromise between better security and better usability.

Is 90 days the right compromise? I don't know.

Re: AWS Secrets Manager – Store, Distribute, and Rotate Credentials Securely

#127

Post by gregmac » Thu, Apr 05, 2018, 4:59 PM UTC

AnimalMuppet wrote: post #126

gregmac wrote: post #125

Earlier quoted context omitted.

What's the rationale behind rotating? I've never really understood this. I'm asking this from a fundamental point-of-view: I realize there are policies and standards that require rotation, but I want to know why those policies say that in the first place. If you have a secret, it's either (potentially) compromised and should be changed immediately, or it's not compromised and thus doesn't need to be changed. If you'r…

You seem to be thinking of a detected compromise, which may have compromised secret X, and your response is to change secret X right now. And I think that's the correct response. But in another sense, all your secrets are potentially compromised, all the time. That is, there are compromises that you don't detect, or don't detect right away. They happen. What should you do? Well, you change the secrets every 90 days,…

If you're talking about secrets used by users (which isn't what AWS Secrets Manager is used for), that is specifically covered by NIST 800-63B [1] which recommends:

* Do not impose other composition rules (e.g. mixtures of different character types) on memorized secrets.

* Do not require that memorized secrets be changed arbitrarily (e.g., periodically) unless there is a user request or evidence of authenticator compromise. (See Section 5.1.1 for additional information).

There's research [2] backing the idea that these policies actually lower security because of the way users act in response.

Systems don't have these problems, so rotating every few seconds doesn't seem to be a big deal. However, if the secret was compromised and you don't know how, you have to assume that the new secret can also be compromised (via the same mechanism), and at the same time there's no reason to assume that an attacker can't make use of it before the next rotation. Thus the rotation is completely ineffective as a means against this unknown attack, so why bother? The energy would be better spent eliminating potential for unknown, undetected attacks.

[1] https://pages.nist.gov/800-63-3/sp800-63b.html#-1021-memoriz...

[2] https://www.ftc.gov/news-events/blogs/techftc/2016/03/time-r...

Re: AWS Secrets Manager – Store, Distribute, and Rotate Credentials Securely

#128

Post by jeremykoerber » Thu, Apr 05, 2018, 5:11 PM UTC

sowbug wrote: post #70

For those of us living in the service-development stone ages, is the idea that a secret-manager service replaces any number of ad-hoc local secret-storage and configuration mechanisms with a single robust mechanism that takes only a single root credential to retrieve all the individual secrets that your service needs? You do still have to figure out a way to securely provide the root credential to your service so tha…

It's a non issue to grant access to a particular group of secrets to a particular server role, assuming the requesting server has an IAM instance profile attached to it. The IAM Policy just describes what secrets the role can access, and then the server simply makes the api call to get the secret.

Having your personal AWS keys compromised should also be a non-issue because of MFA. My keys that I keep in OSX Keychain grant zero permissions, except the ability to assume a more privileged role, but that requires MFA.

FWIW, we're using Parameter Store for secrets and it works great. Given that I just finished that set up just weeks ago, I'm in no rush to jump on the Secrets Manager wagon based on what I'm seeing.

The only problem with both services is the 4k character limit. We have some big cert chains and such that exceed that, so we had to build the tools such that they could put/get a chunked secret. Wasn't too bad though.

Re: AWS Secrets Manager – Store, Distribute, and Rotate Credentials Securely

#129

Post by niklaslindblad » Thu, Apr 05, 2018, 5:27 PM UTC

mlrtime wrote: post #122

niklaslindblad wrote: post #119

Earlier quoted context omitted.

After Amazon EC2 Systems Manager Parameter Store got some much needed love last year (adding KMS encryption and versioning), I wrote a small utility for populating environment variables using a specific Parameter Store prefix: https://github.com/nlindblad/confidential It can either write the environment variables out to a Docker compatible environment file or it can be used as a wrapper to run any executable/script w…

Aren't all your secrets available as environment variables on your instances? If someone has shell access or gets the environment access pragmatically, all your secrets are compromised.

True, but if someone gets shell access as your application user, there is nothing stopping them from impersonating your application or reading any configuration files (including AWS credentials) stored on the file system.

If you are using IAM roles, a HTTP call to http://169.254.169.254/latest/meta-data/iam/security-credent... would give any attacker with shell access the credentials needed to iterate through any secrets in the Parameter Store that the instance has been given access to.

For shell access as another (non-root) user that is not running your application, it is just as safe to rely on environment variables as having configuration files with proper file permissions: https://security.stackexchange.com/questions/14000/environme...

Re: AWS Secrets Manager – Store, Distribute, and Rotate Credentials Securely

#130

Post by ranman » Thu, Apr 05, 2018, 5:42 PM UTC

netvisao wrote: post #98

I am curious as to how it handles those race conditions where a connection is made with the older credentials just after the time the rds master key rotates, or a connection is made with the newer credentials just before the time the rds client key rotates. Short of using two credentials accounts ...

there’s a pretty nifty 4 stage rotation mechanism that solves most, but not all, of those race conditions. More info on it in the post and the docs.