Confidant: an open-source secret management service • Hacker News Board

Re: Confidant: an open-source secret management service

#51

Post by tedmiston » Thu, Nov 05, 2015, 1:20 AM UTC

viraptor wrote: post #36

vangale wrote: post #21

This is the first time I've seen a nicely documented requirements.txt and I like! https://github.com/lyft/confidant/blob/master/requirements.t...

+ for readability, - for precise versions. Good luck receiving critical updates this way...

Pinning precise versions is the best practice for requirements.txt (as opposed to setup.py in a package) (http://nvie.com/posts/pin-your-packages/). Since we have no guarantee that every dependency uses e.g., semantic version, it's the safest way to have reproducible builds across machines.

You can also now list outdated packages with pip if you wanted to upgrade them yourself or test compatibility of new versions. Example for a side project I have laying around:

  $ pip list --outdated
  Django (Current: 1.8.5 Latest: 1.8.6)

  $ pip install --upgrade django
  ...

Re: Confidant: an open-source secret management service

#52

Post by tedmiston » Thu, Nov 05, 2015, 1:25 AM UTC

catshirt wrote: post #40

viraptor wrote: post #36

Earlier quoted context omitted.

+ for readability, - for precise versions. Good luck receiving critical updates this way...

i mean, aren't specified versions crucial for compatibility? what am i missing? on second thought, i guess you are suggesting to specify the major, with an ambiguous minor? but doesn't this require that no package will introduce a breaking change in a minor version? not sure you can rely on that assumption.

Yes, and yes. It's the best practice to have flexible subversions of dependencies (or dependencies of dependencies) when you're shipping a package with setup.py (http://python-packaging-user-guide.readthedocs.org/en/latest...). However in requirements.txt for an app that runs by itself, their approach is the best practice.

Re: Confidant: an open-source secret management service

#53

Post by viraptor » Thu, Nov 05, 2015, 1:38 AM UTC

tedmiston wrote: post #51

viraptor wrote: post #36

Earlier quoted context omitted.

+ for readability, - for precise versions. Good luck receiving critical updates this way...

Pinning precise versions is the best practice for requirements.txt (as opposed to setup.py in a package) ( http://nvie.com/posts/pin-your-packages/ ). Since we have no guarantee that every dependency uses e.g., semantic version, it's the safest way to have reproducible builds across machines. You can also now list outdated packages with pip if you wanted to upgrade them yourself or test compatibility of new versions.…

From a packager's point of view: enforcing specific version results in me having to patch and repackage the service for deployment just because one of its dependencies changed. I understand that keeping track of which project uses what kind of versioning strategy (to apply proper range) is hard - but it's either that or tracking every single point release they do.

I fully agree that deployment should be reproducible and stick to tested versions. But requirements.txt is how you build the software, not how you deploy it. (Unless it is, but then no sympathy from me ;) )

Re: Confidant: an open-source secret management service

#54

Post by njharman » Thu, Nov 05, 2015, 1:50 AM UTC

vangale wrote: post #21

This is the first time I've seen a nicely documented requirements.txt and I like! https://github.com/lyft/confidant/blob/master/requirements.t...

It looks like most of that could be generated. Now I want a tool that looks up that info and outputs annotated requirements file.

Re: Confidant: an open-source secret management service

#55

Post by viraptor » Thu, Nov 05, 2015, 1:54 AM UTC

yeukhon wrote: post #49

viraptor wrote: post #36

Earlier quoted context omitted.

+ for readability, - for precise versions. Good luck receiving critical updates this way...

> + for readability, - for precise versions. Good luck receiving critical updates this way... Real lesson. Authentication failed and no clue why app couldn't start. Developer spent almost two days and realized a new gem version had been released and the new version was not compatible with the current release. Quote developer "fool me once shame on you, fool me twice shame on me." Imagine auto-scale (no image) or inst…

But any critical updates should be API compatible and promoted ASAP. Sure, they need to be verified, but if someone breaks their API with a critical update, that's a separate issue with their release model. If that happens, system packages sometimes fix such incompatibilities (you'll get a new package with patched version instead of a completely new release)

Re: Confidant: an open-source secret management service

#56

Post by e12e » Thu, Nov 05, 2015, 2:33 AM UTC

viraptor wrote: post #55

yeukhon wrote: post #49

Earlier quoted context omitted.

> + for readability, - for precise versions. Good luck receiving critical updates this way... Real lesson. Authentication failed and no clue why app couldn't start. Developer spent almost two days and realized a new gem version had been released and the new version was not compatible with the current release. Quote developer "fool me once shame on you, fool me twice shame on me." Imagine auto-scale (no image) or inst…

But any critical updates should be API compatible and promoted ASAP. Sure, they need to be verified, but if someone breaks their API with a critical update, that's a separate issue with their release model. If that happens, system packages sometimes fix such incompatibilities (you'll get a new package with patched version instead of a completely new release)

From some brief exposure to Plone (and through Plone, Zope) -- I think basing your deployment strategy on "they should" is a bad idea.

Zope originated much of python packaging, through necessity (said tools/tooling has since continued to evolve). Plone still (AFAIK) have massive lists of "known good sets" of dependencies. Often down to the minor version. Because sometimes, what is a bugfix for most consumers of a library trigger latent bugs in other (sets of) libraries.

Yes, you want to fix all the bugs. But sometimes you can't - and then you might need to be explicit in upgrading from 3.2.1 to 3.2.1-bugfix9 rather than making the "leap" to 3.2.2.

Re: Confidant: an open-source secret management service

#57

Post by e12e » Thu, Nov 05, 2015, 2:41 AM UTC

viraptor wrote: post #38

Merkur wrote: post #34

Earlier quoted context omitted.

I did not know such thing is possible. very cool! thx for pointing it out! :) but still..some rhetorical questions: Is KMS based on this concept? Is it's implementation open sourced? Is it verified to run unchanged in the instance I pay? Who is verifying it? Who is paying the one verifying it? Can I meet him / her to drink a coffee and build trust between us? pls, do not misunderstand me. I got more trust in the amaz…

> I did not know such thing is possible. It's theoretically possible, but right now it's too slow to be practical in usual situations. Just storing and single operations are maybe possible. But for example homomorphic AES takes multiple seconds per each block. And that's still AES-128.

Wait, are you saying HE is feasible for 128 bytes of data, if you can allow seconds for de/encryption? That'd be plenty usable for read-at-startup 128 bit api keys for micro-services, disk encryption keys etc?

(Granted, in this context, having a de-crypted disk at run time under a vm would be considered insecure - but still better (for some use cases) than man alternatives)

Re: Confidant: an open-source secret management service

#58

Post by viraptor » Thu, Nov 05, 2015, 2:47 AM UTC

e12e wrote: post #57

viraptor wrote: post #38

Earlier quoted context omitted.

> I did not know such thing is possible. It's theoretically possible, but right now it's too slow to be practical in usual situations. Just storing and single operations are maybe possible. But for example homomorphic AES takes multiple seconds per each block. And that's still AES-128.

Wait, are you saying HE is feasible for 128 bytes of data, if you can allow seconds for de/encryption? That'd be plenty usable for read-at-startup 128 bit api keys for micro-services, disk encryption keys etc? (Granted, in this context, having a de-crypted disk at run time under a vm would be considered insecure - but still better (for some use cases) than man alternatives)

AES has a block size of 128 bits, not bytes. Using the fastest reported method, that would be 2s/block, so 16s/128bytes.

So in practice, probably that's the case - you could use homomorphic encryption for very infrequent operations on KEKs for example.

Re: Confidant: an open-source secret management service

#59

Post by badmadrad » Thu, Nov 05, 2015, 4:29 AM UTC

normanjoyner wrote: post #33

mpdehaan2 wrote: post #27

Earlier quoted context omitted.

I'd say the opposite - the industry is in general being a lot more comfortable with investing in AWS to the point of lockin being acceptable now. More people are treating cloud providers less like a big pool of IP addresses. Not saying it's a good or bad thing, but attend AWS reInvent or something and it's pretty easy to percieve huge uptake. If you adopt extra management stacks running on or against your cloud you l…

That's an interesting perspective and observation. I've always found lock-in to be a bit scary, and I've heard this echoed by a number of colleagues & companies. No one can deny the usefulness of tools provided by hosting providers, namely AWS; however, to me, architecting a system in such a way where you are literally unable to jump ship when the need arises (compliance, cost, etc) is unacceptable. With hybrid / mul…

I would tend to agree here to some extent. I think you have to be careful what you lock into. We use RDS and Elasticache but if push came to shove we could easily spin up our own db instances elsewhere or use some other managed service. That being said, locking into something as integral about how you handle your secrets is a big decision. One that when you make, you must be sure you are going to be in AWS for long haul. The reason is how you handle your secrets will touch how everything in your infrastructure is deployed. Migrating to a new system is not easy. (Trust me I am trying to migrate to Vault) For this, I would prefer something less rigid like Vault but I think this tool seems pretty cool.

Re: Confidant: an open-source secret management service

#60

Post by e12e » Thu, Nov 05, 2015, 4:44 AM UTC

viraptor wrote: post #58

e12e wrote: post #57

Earlier quoted context omitted.

Wait, are you saying HE is feasible for 128 bytes of data, if you can allow seconds for de/encryption? That'd be plenty usable for read-at-startup 128 bit api keys for micro-services, disk encryption keys etc? (Granted, in this context, having a de-crypted disk at run time under a vm would be considered insecure - but still better (for some use cases) than man alternatives)

AES has a block size of 128 bits, not bytes. Using the fastest reported method, that would be 2s/block, so 16s/128bytes. So in practice, probably that's the case - you could use homomorphic encryption for very infrequent operations on KEKs for example.

Well, actually I was wrong twice, so it cancels out ;-) 128 bits is enough for many things. Eg. 128 bit keys. Too little for an ECC secret key, but not by much.