Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Entropy, a CLI that scans files to find high entropy lines (might be secrets)

github.com

Post reply on HN
Print view View original

21–30 of 141 posts

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#21

Post by j16sdiz »

__MatrixMan__ wrote: post #16
coppsilgold wrote: post #7

Note that in an adversarial setting this will only be effective against careless opponents. If you properly encode your secret it will have the entropy of its surroundings. For example you can hide a string of entropy (presumably something encrypted) in text as a biased output of an LLM. To recover it you would use the same LLM and measure deviations from next-token probabilities. This will also fool humans examining…

I imagine a social media site full of bots chatting about nonsense. Hidden in the nonsense are humans chatting about different nonsense. This way, server costs get paid for by advertisers, but its really only bots that see the ads anyway.

It's called Twitter.

It's no nonsense, just catvideo and porns.

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#22

Post by thephyber »

textninja wrote: post #10
coppsilgold wrote: post #7

Note that in an adversarial setting this will only be effective against careless opponents. If you properly encode your secret it will have the entropy of its surroundings. For example you can hide a string of entropy (presumably something encrypted) in text as a biased output of an LLM. To recover it you would use the same LLM and measure deviations from next-token probabilities. This will also fool humans examining…

What you described sounds like a very cool idea - LLM-driven text steganography, basically - but intentional obfuscation is not the problem this tool is trying to solve. To your point about secrets with entropy similar to the surrounding text, however, I wonder if this can pick up BIP39 Seed Phrases or if whole word entropy fades into the background.

The LLM adds no value here. Procedural generation in a loop until some fitness function (perhaps a frequency analysis metric) is satisfied.

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#24

Post by buildbot »

dragonwriter wrote: post #14
buildbot wrote: post #11

Earlier quoted context omitted.

In general (for those unaware) this is called stenography. You can hide an image in the lower bits of another image for example too.

Steganography ; stenography is completely different.

Thanks

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#26

Post by daemonologist »

icapybara wrote: post #23

Gonna have to explain how a “high entropy line” is calculated and why it might be secrets.

Entropy of information is basically how well it can be compressed. Random noise usually doesn't compress much at all and thus has high entropy, whereas written natural language can usually be compressed quite a bit. Since many passwords and tokens will be randomly generated or at least nonsense, looking for high entropy might pick up on them.

This package seems to be measuring entropy by counting the occurrences of each character in each line, and ranking lines with a high proportion of repeated characters as having low entropy. I don't know how closely this corresponds with the precise definition. Source: https://github.com/EwenQuim/entropy/blob/f7543efe130cfbb5f0a...

More: https://en.wikipedia.org/wiki/Entropy_(information_theory)

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#27

Post by hamasho »

I didn't know what entropy means in software, so here's the definition[0]:

----

  Software entropy is a measure of the disorder or complexity of a software system. It is a natural tendency for software entropy to increase over time, as new features are added and the codebase becomes more complex.

  High entropy in software development means that the code is difficult to understand, maintain, and extend. It is often characterized by:

      Duplicated code: The same code or functionality is repeated in multiple places, which can make it difficult to find and fix bugs.
      Complex logic: The code is difficult to follow and understand, which can make it difficult to add new features or fix bugs without introducing new ones.
      Poor documentation: The code is not well-documented, which can make it difficult for new developers to understand and contribute to the codebase.
      Technical debt: The code has been patched and modified over time without proper refactoring, which can lead to a tangled and cluttered codebase.

  Low entropy in software development means that the code is well-organized, easy to understand, and maintain. It is often characterized by:

      Well-designed architecture: The code is structured in a logical way, with clear separation of concerns.
      Consistent coding style: The code follows a consistent coding style, which makes it easy to read and understand.
      Comprehensive documentation: The code is well-documented, with clear explanations of the code's purpose and functionality.
      Minimal technical debt: The code has been refactored regularly to remove technical debt, which makes it easy to add new features and fix bugs without introducing new ones.
[0] https://www.kisphp.com/python/high-and-low-entropy-in-softwa...

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#28

Post by itemize »

hamasho wrote: post #27

I didn't know what entropy means in software, so here's the definition[0]: ---- Software entropy is a measure of the disorder or complexity of a software system. It is a natural tendency for software entropy to increase over time, as new features are added and the codebase becomes more complex. High entropy in software development means that the code is difficult to understand, maintain, and extend. It is often chara…

thanks for the search. this is textual entropy however, I am not sure if definition is applicable

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#29

Post by kmoser »

thsksbd wrote: post #6

This is very cool, but I have a thought - I see this as a last line of defense, and I am concerned that this would this give a false sense of security leading people to be more reckless with secrets.

You could make the same argument for any tool that does not provide high security. In fact security is layered, and no single tool should be relied upon to be your one security tool. You said as much yourself: "I see this as a last line of defense," but I don't see how you conclude that this would inherently cause people to be more reckless with secrets.

Re: Entropy, a CLI that scans files to find high entropy lines (might be secrets)

#30

Post by eviks »

spullara wrote: post #17
__MatrixMan__ wrote: post #16

Earlier quoted context omitted.

I imagine a social media site full of bots chatting about nonsense. Hidden in the nonsense are humans chatting about different nonsense. This way, server costs get paid for by advertisers, but its really only bots that see the ads anyway.

if the ads aren't effective people won't buy them

People have been buying ineffective ads since the invention of ads
Post reply on HN