Live data from Hacker News

Entropy, a CLI that scans files to find high entropy lines (might be secrets)

github.com

41–50 of 141 posts

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#41
post #21

Earlier quoted context omitted.

I imagine a social media site full of bots chatting about nonsense. Hidden in the nonsense are humans chatting about different nonsense. This way, server costs get paid for by advertisers, but its really only bots that see the ads anyway.

It's called Twitter. It's no nonsense, just catvideo and porns.

Hmm yes, sensical things those.

Are you proposing that they're really only posted as a medium for encoding something else that we're not privy to? If so, somebody took my idea.

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#42
post #37

Is there any good posts about the use of entropy for tasks like that? I am wondering for quite some time of how do people actually use it and if it is any effective, but never actually got to investigating the problem myself. First of all, how to define "entropy" for text is a bit unclear in the first place. Here it's as simple as `-Sum(x log(x))` where x = countOccurences(char) / len(text). And that raises a lot of…

Entropy is a measure of complexity or disorder of a signal. The interesting part is that the disorder is with respect to the proper basis or dictionary. Something can look complex in one encoding but be low entropy in the right encoding. You need to know the right basis, or figure it out from the context, to accurately determine the entropy of a signal. A much stronger way of building a tool like the OP is to have a few pre-computed dictionaries for a range of typical source texts (source code, natural language), then encode the string against each dictionary, comparing the compressibility of the string. A high entropy string like a secret will compress poorly against all available dictionaries.

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#43
post #34

Earlier quoted context omitted.

Of course, this heuristic fails for weak passwords. And it fails for passphrases like 'correct battery horse staple', which have a large enough total entropy to be good passwords, but have a low entropy per character.

4 diceware words is hardly a good password. It's ~51 bits of entropy, about the same as 8 random ascii symbols. It could be trivially cracked in less than an hour. Your average variable name assigned to the result of an object name with a method name called with a couple parameter names has much more entropy.

Just imagine my example used 8 words.

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#44
post #6

This is very cool, but I have a thought - I see this as a last line of defense, and I am concerned that this would this give a false sense of security leading people to be more reckless with secrets.

Welcome to information security :)

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#46
post #37

Is there any good posts about the use of entropy for tasks like that? I am wondering for quite some time of how do people actually use it and if it is any effective, but never actually got to investigating the problem myself. First of all, how to define "entropy" for text is a bit unclear in the first place. Here it's as simple as `-Sum(x log(x))` where x = countOccurences(char) / len(text). And that raises a lot of…

[deleted]

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#47

Earlier quoted context omitted.

4 diceware words is hardly a good password. It's ~51 bits of entropy, about the same as 8 random ascii symbols. It could be trivially cracked in less than an hour. Your average variable name assigned to the result of an object name with a method name called with a couple parameter names has much more entropy.

If you can crack a single 52bit password in an hour, that's suggesting you can crack a 40bit password every second. That's 1 trillion hashes per second.

Salts and timeouts made that password cracking technique obsolete anyways.

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#48
post #30

Earlier quoted context omitted.

if the ads aren't effective people won't buy them

People have been buying ineffective ads since the invention of ads

Not really, advertising is really the only field of human endeavour that is both data-driven and results-oriented.

(Doesn't still stop smart people from committing fraud, but that is a different story.)

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#49
post #34

Earlier quoted context omitted.

Of course, this heuristic fails for weak passwords. And it fails for passphrases like 'correct battery horse staple', which have a large enough total entropy to be good passwords, but have a low entropy per character.

4 diceware words is hardly a good password. It's ~51 bits of entropy, about the same as 8 random ascii symbols. It could be trivially cracked in less than an hour. Your average variable name assigned to the result of an object name with a method name called with a couple parameter names has much more entropy.

So you do random capital words, random punctuation and add a number somewhere and you’re at 60. Add more for whatever threat model you’re trying to be secure against.

https://beta.xkpasswd.net/

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#50

See also: - trufflehog: https://github.com/trufflesecurity/trufflehog - detect-secrets: https://github.com/Yelp/detect-secrets - semgrep secrets: https://semgrep.dev/products/semgrep-secrets -- (Paid, but may be included in existing licenses in some cases

Also see PyWhat for both interesting strings and secrets https://github.com/bee-san/pyWhat
Post reply on HN