I'm a scientist. Information theory always felt curious to me because it adopted terminology and concepts similar to that from statistical mechanics but in practice was always much more difficult due to what could be considered what was "random" vs. what was non-random. After sitting back and thinking about it, what is considered "random" of course is a statement of the probability distribution for the set under consideration. For info theory, that set is some set of strings (say passwords) which is really culturally and historically contigent, while in physics land, the set is microstates that determine macrostates, for which the degeneracy of a macrostate depends on the hamiltonian, full stop. I think mathematically, of course the statements you make are similar (hence why you apply the same prob theory to both) but the systems I study are comparatively easier, while really, the underlying probability distribution for strings is really hard to know in practice because it essentially depends on human history and culture up to that point. For example, in a universe without English, English words (say one-to-oned to a discrete set, so strings of positive integers less than 26 + 10 (including decimal numbers)) would be random. In fact, a universe without that particular Chinese IME, if it was done somewhat differently, then ji32k7au4a83 could be random.
It's just interesting to me, another reminder that physics is just that much more easier than anything else.