This reminds me of the program 'ent' (which I have used for a very long time)
Entropy, a CLI that scans files to find high entropy lines (might be secrets)
81–90 of 141 posts
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#82Earlier quoted context omitted.
The tool found "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890" in our codebase as a high entropy line :)
Well, it is...
I mean, it certainly has a low Kolmogorov complexity (which is what I would really want to be measuring somehow for this tool... note that I am not claiming that is possible: just an ideal); I am unsure whether how that affects the related bounds on Shannon entropy, though.
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#83Earlier quoted context omitted.
I was imagining the message encoded in clear text, not encrypted form, because given the lengths required to coordinate protocol, keys, weights, and so on, I assumed there would be more efficient ways to disguise a message than a novel form of steganography. As such, I approached it as a toy problem, and considered detection by savvy parties to be a feature, not a bug; I imagined something more like a pirate broadcas…
> I was imagining the message encoded in clear text, not encrypted form, [...] I was considering that, but I came to the conclusion that it would be an exceedingly poor choice. Steganography is there to hide that a message has been sent at all. If you make it do double duty as a poor-man's encryption, you are going to have a bad time. > As such, I approached it as a toy problem, and considered detection by savvy part…
> If you make it do double duty as a poor-man's encryption, you are going to have a bad time.
For the serious use cases you evidently have in mind, yes, it's folly to have it do double duty, but at the end of the day steganography is an obfuscation technique orthogonal to encryption, so the question of whether to use encryption or not is a nuanced one. Anyhow, I don't think it's fair to characterize this elaborate steganography tech as a poor-man's encryption — LLM tokens are expensive!
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#84Interesting. If I had to do this, I would have done something like perl -lne 'next unless $_; $z = qx(echo "$_" | gzip | wc -c); printf "%5.2f %s\n", $z/length($_), $_' on the principle that high entropy means it compresses badly. However, that uses each line as the dictionary, rather than the entire file, so it has a little trouble with very short lines which compress badly. It did react to this line return map { $_…
I guess you could take all lines in the file except the one you're testing and measure the filesize, then add the line and measure again. The delta should then be more fair. You could even do this by concatenating all code files and then testing line by line across the entire repo, but that would probably be too slow.
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#85Earlier quoted context omitted.
> I was imagining the message encoded in clear text, not encrypted form, [...] I was considering that, but I came to the conclusion that it would be an exceedingly poor choice. Steganography is there to hide that a message has been sent at all. If you make it do double duty as a poor-man's encryption, you are going to have a bad time. > As such, I approached it as a toy problem, and considered detection by savvy part…
> If you make it do double duty as a poor-man's encryption, you are going to have a bad time. For the serious use cases you evidently have in mind, yes, it's folly to have it do double duty, but at the end of the day steganography is an obfuscation technique orthogonal to encryption, so the question of whether to use encryption or not is a nuanced one. Anyhow, I don't think it's fair to characterize this elaborate st…
> Anyhow, I don't think it's fair to characterize this elaborate steganography tech as a poor-man's encryption — LLM tokens are expensive!
I guess it's a "rich fool's encryption".
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#86Interesting. If I had to do this, I would have done something like perl -lne 'next unless $_; $z = qx(echo "$_" | gzip | wc -c); printf "%5.2f %s\n", $z/length($_), $_' on the principle that high entropy means it compresses badly. However, that uses each line as the dictionary, rather than the entire file, so it has a little trouble with very short lines which compress badly. It did react to this line return map { $_…
I would use a better compressor than gzip but I have done this trick several times.
xz or zstd may be better choices, or you can look at Hutter Prize [1] winners for best compression and therefore best entropy estimate.
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#87Earlier quoted context omitted.
Unfortunately I beg to differ. I worked for several companies where we the management clearly saw that the results were very poor (for Facebook ads, for example) but continued to invest because there is a defined budget for it and so on. It was like this last year and 20 years ago.
these companies should be outcompeted by firms that don't blow a million dollars a month paying out to click fraudsters but alas the market is not perfectly competitive is it a cargo cult? it works for coca cola so maybe if we just spend a little more we'll see returns...
Yes, I feel it might be cargo cult, at least in part. The argument I usually heard was that "But other companies are doing that, too".
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#88Earlier quoted context omitted.
If you're trying to improve the security of your product by running random binaries from the Internet you're going to have a bad time
That's how most people run compilers
This is argumentum ad absurdum - there is a reason why trusting your kernel and compiler is a reasonable compromise, even though there might be security issues in them, but random pieces of software downloaded from the Internet is not.
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#89Another way to do this would be to compress the file and compare the compressed size to the uncompressed size.
Encrypted files do not compress well compared to code, I saw a phd thesis that postulated an inverse ratio of compression efficiency to performance data mining, this would be the opposite
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#90xkcd.com/936/