Live data from Hacker News

Entropy, a CLI that scans files to find high entropy lines (might be secrets)

github.com

131–140 of 141 posts

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#131
Neat tool.

Would be cool if this CLI could have a flag to read .gitignore and exclude all of the contents automatically.

Also it might be cool to have different strategies for detecting secrets, e.g. Kolmogorov complexity as other comments have noted.

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#132
post #56

Earlier quoted context omitted.

Bots do click in real ad fraud, so your moved goalpost isn't all that solid

sorry, conversions is really what I meant. if the bots are also buying the stuff then it would work.

I'm not really concerned with whether it would work out in a way that was beneficial for the ad network or their customers. Either they figure out a way to make it work such that they continue to be a useful pipe, or they don't and then maybe they'll have to shut down and find something more productive to do with their time. We'll have done the public a service either way.

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#133
post #55

Interesting. If I had to do this, I would have done something like perl -lne 'next unless $_; $z = qx(echo "$_" | gzip | wc -c); printf "%5.2f %s\n", $z/length($_), $_' on the principle that high entropy means it compresses badly. However, that uses each line as the dictionary, rather than the entire file, so it has a little trouble with very short lines which compress badly. It did react to this line return map { $_…

Are there any command-line tools for zip or similar that allow you to predefine a dictionary over one or more files, and then use that dictionary to compress small files? Which would require the dictionary as a separate input when decompressing, of course?

zstd supports shared dictionaries easily.

It also has vastly superior compression and performance, compared to gzip, even without.

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#134

Why would I need to install go to run this tool? I thought one advantage of go was that devs could just distribute a single binary file that works...

I'd love to have it on homebrew but my PR is denied so I'll have to create my own brew tap or convince them to accept it. I'll also create a docker image. I just didn't expect this much popularity so the repo isn't 100% ready te be honest

The docker image is nothing more than “FROM scratch” and then copying in the statically linked binary. If there are 0 other dependencies, I think it would be equally easy to distribute the binary through GitHub releases. There is no need for brew.

If people want to run it isolated, the docker image is of course still a nice to have and docker hub is a convenient distribution mechanism. At the same time, an equivalent image can equally easily be created by the consumer.

Personally, due to trust, I would anyway still build it from source and run in a container, to be on the safe side.

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#135

Earlier quoted context omitted.

If you can crack a single 52bit password in an hour, that's suggesting you can crack a 40bit password every second. That's 1 trillion hashes per second.

350B H/s was achieved in 2012 on consumer hardware. That's over 12 years ago, and several lifetimes of GPU improvements ago. 4 diceware words is simply not appropriate for anything remotely confidential, and it is bad for the community to pretend otherwise. https://theworld.com/~reinhold/dicewarefaq.html

If you read the sources, that's 350B _sha1_ hashes per second... While you can't be sure what hash system is being used for your passwords, any respectable system using a modern password hash is not even close to being that fast. OWASP's recommended 600000 rounds of pbkdf2 performs 1.2 million sha2 block rounds IIRC. If we assume that sha1 and sha2 are equivalent in performance, then you're looking at only 290,000 password attempts a second.

If the password system uses argon2 with a high memory requirement, you're in an even better position

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#136

Earlier quoted context omitted.

350B H/s was achieved in 2012 on consumer hardware. That's over 12 years ago, and several lifetimes of GPU improvements ago. 4 diceware words is simply not appropriate for anything remotely confidential, and it is bad for the community to pretend otherwise. https://theworld.com/~reinhold/dicewarefaq.html

If you read the sources, that's 350B _sha1_ hashes per second... While you can't be sure what hash system is being used for your passwords, any respectable system using a modern password hash is not even close to being that fast. OWASP's recommended 600000 rounds of pbkdf2 performs 1.2 million sha2 block rounds IIRC. If we assume that sha1 and sha2 are equivalent in performance, then you're looking at only 290,000 pa…

Certainly if we assume the system under question was deigned with heightened security in mind, we will determine ourselves to be in a more secure system.

But go on, use a 52 bit password – see what I care. But don't come crying to me when an institution with the smallest amount of funding was able to crack your vault.

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#137

Earlier quoted context omitted.

does the stated purpose of the tool influence whether or not you can trust it?

I think that question is a little backwards. Certain tools are more likely to be used by people working in spaces where they should/must be less trusting. If there was a tool (there is) to scan my platform deployment against some NCSC/NSA guidance for platform security, and I wanted to use it, I'm likely operating in a space that should consider being cautious about running random tools I find on the internet.

right, but in that scenario I'd assume you'd also want to take a look at your ostensibly unrelated tools

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#138

Earlier quoted context omitted.

does the stated purpose of the tool influence whether or not you can trust it?

If you're trying to improve the security of your product by running random binaries from the Internet you're going to have a bad time

also if you're not trying to improve the security of your product by running random binaries from the internet. I'm concerned at the inability to separate the concepts of "what it does" and "what it says it does".

The idea that whether or not it needs scrutiny is impacted by your goals with the software is... creative

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#139
post #55

Interesting. If I had to do this, I would have done something like perl -lne 'next unless $_; $z = qx(echo "$_" | gzip | wc -c); printf "%5.2f %s\n", $z/length($_), $_' on the principle that high entropy means it compresses badly. However, that uses each line as the dictionary, rather than the entire file, so it has a little trouble with very short lines which compress badly. It did react to this line return map { $_…

I use Yelp's secret scanner as a pre-commit hook, they have pretty easy setup be pre-commit's install mechanism

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#140
post #55

Interesting. If I had to do this, I would have done something like perl -lne 'next unless $_; $z = qx(echo "$_" | gzip | wc -c); printf "%5.2f %s\n", $z/length($_), $_' on the principle that high entropy means it compresses badly. However, that uses each line as the dictionary, rather than the entire file, so it has a little trouble with very short lines which compress badly. It did react to this line return map { $_…

Are there any command-line tools for zip or similar that allow you to predefine a dictionary over one or more files, and then use that dictionary to compress small files? Which would require the dictionary as a separate input when decompressing, of course?

[deleted]
Post reply on HN