Earlier quoted context omitted.
The tool found "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890" in our codebase as a high entropy line :)
Well, it is...
…a very verbose way to match alphanumeric characters :-)
Entropy, a CLI that scans files to find high entropy lines (might be secrets)
101–110 of 141 posts
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#102It would be useful if it also trawled through the full git history of the project - a secret could have been checked in and later removed, but still exist in the history.
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#103Earlier quoted context omitted.
4 diceware words is hardly a good password. It's ~51 bits of entropy, about the same as 8 random ascii symbols. It could be trivially cracked in less than an hour. Your average variable name assigned to the result of an object name with a method name called with a couple parameter names has much more entropy.
Just imagine my example used 8 words.
But it didn't. It perpetuated the exceedingly common myth that 52 bits is somehow enough. This has been considered bad practice for well over a decade now. https://theworld.com/~reinhold/dicewarefaq.html
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#104Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#105See also: - trufflehog: https://github.com/trufflesecurity/trufflehog - detect-secrets: https://github.com/Yelp/detect-secrets - semgrep secrets: https://semgrep.dev/products/semgrep-secrets -- (Paid, but may be included in existing licenses in some cases
noseyparker is another good one: https://github.com/praetorian-inc/noseyparker
I think these solutions are all much better for finding secrets than something naive based on entropy. Yes, entropy is more general but these are well established tools that have been through the fire of many, many data sets.
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#106Earlier quoted context omitted.
If you can crack a single 52bit password in an hour, that's suggesting you can crack a 40bit password every second. That's 1 trillion hashes per second.
Salts and timeouts made that password cracking technique obsolete anyways.
Only for online access. Offline access is still a thing, and in no way "obsolete".
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#107Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#108Why would I need to install go to run this tool? I thought one advantage of go was that devs could just distribute a single binary file that works...
Because it's a security tool so trusting a binary upfront defeats the purpose. With source you at least have the option to inspect what it really does.
Uh? OP just released a docker image and wants to release a homebrew thingy. Even assuming that was you say is somehow sensible, it's not the reason, no. You're just grasping at straws.
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#109Interesting. If I had to do this, I would have done something like perl -lne 'next unless $_; $z = qx(echo "$_" | gzip | wc -c); printf "%5.2f %s\n", $z/length($_), $_' on the principle that high entropy means it compresses badly. However, that uses each line as the dictionary, rather than the entire file, so it has a little trouble with very short lines which compress badly. It did react to this line return map { $_…
I learned Go many years ago doing some advent of code problems. As I solved each problem, my housemate pestered me for a look and then rewrote my solutions (each needing 10-50 lines of go) into Ruby one-liners. All the while making fun of Go and my silly programs. I wasn’t intending to, but I ended up learning a lot Ruby that night too.
Thankyou for continuing the tradition.
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#110Is there any good posts about the use of entropy for tasks like that? I am wondering for quite some time of how do people actually use it and if it is any effective, but never actually got to investigating the problem myself. First of all, how to define "entropy" for text is a bit unclear in the first place. Here it's as simple as `-Sum(x log(x))` where x = countOccurences(char) / len(text). And that raises a lot of…
Entropy of a particular string isn't a rigorous mathematical idea, since by definition the string which is known can only take one value, the "entropy" is therefore zero bits. The reason why we can distinguish non-random data from random is that only a small subset of all possible states are considered useful for humans, and since we have an idea what that subset looks like, we can try to estimate what process was used to generate a particular string.
There are of course statistical tests like https://en.wikipedia.org/wiki/Diehard_tests, which are good enough for distinguishing low entropy and high entropy data, but current pseudo-random number generators have no problem passing all of those, even though their actual "entropy" is just the seed plus approximate the complexity of the algorithm.