Earlier quoted context omitted.
zero clicks is a little different
Bots do click in real ad fraud, so your moved goalpost isn't all that solid
sorry, conversions is really what I meant. if the bots are also buying the stuff then it would work.
Entropy, a CLI that scans files to find high entropy lines (might be secrets)
121–130 of 141 posts
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#122Interesting. If I had to do this, I would have done something like perl -lne 'next unless $_; $z = qx(echo "$_" | gzip | wc -c); printf "%5.2f %s\n", $z/length($_), $_' on the principle that high entropy means it compresses badly. However, that uses each line as the dictionary, rather than the entire file, so it has a little trouble with very short lines which compress badly. It did react to this line return map { $_…
I would use a better compressor than gzip but I have done this trick several times. xz or zstd may be better choices, or you can look at Hutter Prize [1] winners for best compression and therefore best entropy estimate. [1] http://prize.hutter1.net/
> best compression and therefore best entropy estimate
That's a good point. But the Hutter Prize is for compressing a 1 GB file. On inputs as short as a line of code, gzip doesn't do so badly. For a longer line:
$ INPUT=' bool isRegPair() const { return kind() == RegisterPair || kind() == LateRegisterPair || kind() == SomeLateRegisterPair; }'
$ echo "$INPUT" | gzip | wc -c
95
$ echo "$INPUT" | bzip2 | wc -c
118
$ echo "$INPUT" | xz -F xz | wc -c
140
$ echo "$INPUT" | xz -F lzma | wc -c
97
$ echo "$INPUT" | zstd | wc -c
92
$ INPUT=' ASSERT(regHi().isGPR());'
$ echo "$INPUT" | gzip | wc -c
48
$ echo "$INPUT" | bzip2 | wc -c
73
$ echo "$INPUT" | xz -F xz | wc -c
92
$ echo "$INPUT" | xz -F lzma | wc -c
51
$ echo "$INPUT" | zstd | wc -c
46Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#123Interesting. If I had to do this, I would have done something like perl -lne 'next unless $_; $z = qx(echo "$_" | gzip | wc -c); printf "%5.2f %s\n", $z/length($_), $_' on the principle that high entropy means it compresses badly. However, that uses each line as the dictionary, rather than the entire file, so it has a little trouble with very short lines which compress badly. It did react to this line return map { $_…
Are there any command-line tools for zip or similar that allow you to predefine a dictionary over one or more files, and then use that dictionary to compress small files?
Which would require the dictionary as a separate input when decompressing, of course?
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#124I transcend this problem by making all my database passwords 'abcd'
Username: postgres
Password: postgres
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#125Thank you DrJones for asking what a high entropy string is several years ago[0] and linking to a good article on it.[1]
[0] https://news.ycombinator.com/item?id=13304641
[1] https://www.splunk.com/en_us/blog/security/random-words-on-e...
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#126Interesting. If I had to do this, I would have done something like perl -lne 'next unless $_; $z = qx(echo "$_" | gzip | wc -c); printf "%5.2f %s\n", $z/length($_), $_' on the principle that high entropy means it compresses badly. However, that uses each line as the dictionary, rather than the entire file, so it has a little trouble with very short lines which compress badly. It did react to this line return map { $_…
Are there any command-line tools for zip or similar that allow you to predefine a dictionary over one or more files, and then use that dictionary to compress small files? Which would require the dictionary as a separate input when decompressing, of course?
gzip (or really DEFLATE) does actually come with a small predefined dictionary (the "fixed Huffman codes" in the RFC) which is somewhat optimised for latin letters in UTF-8, but I have not verified that this is indeed what ends up being used when compressing individual lines of source code.
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#127Earlier quoted context omitted.
If you’re looking for a rigorous mathematical idea, what people are trying to measure is the Kolmogorov complexity of the code. Measuring the compressed length is a rough estimate of that value. https://en.m.wikipedia.org/wiki/Kolmogorov_complexity
Yes, although (and here my understanding of Kolmogorov complexity ends) it still depends heavily on the choice of language and it seems to me like "aaaaaaaaa" is only less complex than "pSE+4z*K58" due to assuming a sane, human-centric language which is very different from the "average" of all possible languages. Which then leads me to wonder how to construct an adversarial turing-complete language which has unintuit…
Kolmogorov complexity conventionally refers to the Turing machine as the base for implementation. This indeed makes repeated letters significantly less complex than that other string. (If you want intuition for how much code is needed to do something on a Turing machine, learn and play around a bit with Brainfuck. It's actually quite nice for that.)
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#128Earlier quoted context omitted.
Reminds me of https://xkcd.com/936/ I think "correct horse battery staple" has a low entropy, since it is just ordinary looking words (strings).
A quick Google search suggests English has about 10 bits of entropy per word. Having a long password like that can still have high total entropy I suppose, but it has a low entropy density .
Maybe 10 bits is the average over the dictionary – which is what matters here, but over normal text it is significantly less. Our best current estimation for relatively high-level text (texts published by the EU) is 6 bits per word[1].
However, as our methods of predicting text improve, this number is revised down. LLMs ought to have made a serious dent in it, but I haven't looked up any newer results.
Anyway, all of this to say is that which words are chosen matters, but how they are put together matters perhaps more.
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#129Is there any good posts about the use of entropy for tasks like that? I am wondering for quite some time of how do people actually use it and if it is any effective, but never actually got to investigating the problem myself. First of all, how to define "entropy" for text is a bit unclear in the first place. Here it's as simple as `-Sum(x log(x))` where x = countOccurences(char) / len(text). And that raises a lot of…
Entropy is a measure of complexity or disorder of a signal. The interesting part is that the disorder is with respect to the proper basis or dictionary. Something can look complex in one encoding but be low entropy in the right encoding. You need to know the right basis, or figure it out from the context, to accurately determine the entropy of a signal. A much stronger way of building a tool like the OP is to have a…
I only briefly browsed the code, but this seems to be roughly what yelp/detect-secrets does.
Anyway, that doesn't really answer my question. To summarize answers in this thread, I think PhilipRoman has captured the essence of it: strictly speaking, the idea of entropy of a known string is nonsense. So, as I suspected, information theory definition isn't meaningfully applicable to the problem. And as other commenters like you mentioned, what we are really trying to measure is basically Kolmogorov complexity, which, strictly speaking, is incomputable, but measuring the compression rate for some well-known popular compression algorithm (allegedly) seems to be good enough estimate, empirically.
But I think it's still an interesting linguistic question. Meaningful or not, but it's well defined: so does it appear to work? Are there known constants for different kinds of text for any of these (or other) metrics? I would suspect this should have been explored already, but neither me, nor anybody in this thread apparently has ever stumbled upon such article.
Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)
#130Earlier quoted context omitted.
If you’re looking for a rigorous mathematical idea, what people are trying to measure is the Kolmogorov complexity of the code. Measuring the compressed length is a rough estimate of that value. https://en.m.wikipedia.org/wiki/Kolmogorov_complexity
Yes, although (and here my understanding of Kolmogorov complexity ends) it still depends heavily on the choice of language and it seems to me like "aaaaaaaaa" is only less complex than "pSE+4z*K58" due to assuming a sane, human-centric language which is very different from the "average" of all possible languages. Which then leads me to wonder how to construct an adversarial turing-complete language which has unintuit…
> due to assuming a sane, human-centric language
There’s no requirement that the K-complexity is measured in a human centric language. Arguably all compression formats are languages too, which can be executed to produce the decompressed result. They are not designed to be human centric at all, and yet they do a surprisingly decent job at providing an estimate (well, upper bound) on Kolmogorov complexity. - As we can see in this program.