Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Shannon: Tinkerer, Prankster, and Father of Information Theory (2016)

spectrum.ieee.org

Post reply on HN
Print view View original

21–30 of 38 posts

Re: Claude Shannon: Tinkerer, Prankster, and Father of Information Theory (2016)

#21

Post by sig-reduction »

TeMPOraL wrote: post #5
op03 wrote: post #3

From what I understand, before Shannon showed up no one knew how to calculate the Channel Capacity of a medium and how to deal with noise in the medium. So people just yelled louder or yelled repeatedly to get things across with better reliability. And when channel capacity was exceeded with everyone yelling too much, no one knew it had exceeded and lot of energy and time was wasted with increasing errors in the syst…

> Doesn't that feel like a repeat with whats happening in social media and news media these days? Or is it just different things. Maybe. It can feel like this, but applying mathematical models to social phenomena is tricky. You have to map formal parameters to fuzzy, poorly understood factors. How do you define, precisely, what is a "channel" on a social network? What is "signal", and what is "noise"? People can and…

I think OP has a good point. Although language if fuzzy, so are "binary" signals once you drill down to the physical layer. The fuzziness of language goes away if you allow yourself to assume an idealized language embedding model. Similar ideas are in an epsilon ball around the embedding of some concrete linguistic instantiation of the idea.

An example of a binary signal: "It is unambiguously true that a lab leak did not occur in Wuhan." The spread of this binary signal could be measured using the technique described above. A BERT embedding may suffice.

The channels are just tweets, DMs, and Vox articles.

Re: Claude Shannon: Tinkerer, Prankster, and Father of Information Theory (2016)

#22

Post by 2snakes »

1cvmask wrote: post #20
harveywi wrote: post #14

Earlier quoted context omitted.

> At the banquet, the meeting’s organizers somehow persuaded Shannon to address the audience. He spoke for a few minutes and then—fearing that he was boring his audience, he recalled later—pulled three balls out of his pockets and began juggling. Shannon was famous for his entropy.

The alleged origins of the naming of entropy: When Shannon first derived his famous formula for information, he asked von Neumann what he should call it and von Neumann replied “You should call it entropy for two reasons: first because that is what the formula is in statistical mechanises but second and more important, as nobody knows what entropy is, whenever you use the term you will always be at an advantage! From…

Yeah, I would characterize it as a mathematical theory of -symbols- and that information has different definitions. However, the argument can be made that symbolic representation is a principle of reality.

Re: Claude Shannon: Tinkerer, Prankster, and Father of Information Theory (2016)

#23

Post by steve76 »

op03 wrote: post #3

From what I understand, before Shannon showed up no one knew how to calculate the Channel Capacity of a medium and how to deal with noise in the medium. So people just yelled louder or yelled repeatedly to get things across with better reliability. And when channel capacity was exceeded with everyone yelling too much, no one knew it had exceeded and lot of energy and time was wasted with increasing errors in the syst…

Information, as in interaction of subatomic particles. No analogy to the bad habits and lack of culture to humans on a propped up bailed out market.

Re: Claude Shannon: Tinkerer, Prankster, and Father of Information Theory (2016)

#24

Post by mycologos »

sillysaurusx wrote: post #19

Shannon was cursed by being the inventor of information theory. It seems like this wonderful thing; who wouldn’t want to invent information theory? But as an inventor, you’ll never top it. You’ll forever live in your own shadow. And for someone that enjoys inventing things, that’s... not a happy thing.

Nothing I've read about Shannon suggests that he felt "cursed" in his later life. Is there some reason to think otherwise, or is this just assuming that anybody who does something big early feels sad about it later?

Re: Claude Shannon: Tinkerer, Prankster, and Father of Information Theory (2016)

#27

Post by gugagore »

op03 wrote: post #3

From what I understand, before Shannon showed up no one knew how to calculate the Channel Capacity of a medium and how to deal with noise in the medium. So people just yelled louder or yelled repeatedly to get things across with better reliability. And when channel capacity was exceeded with everyone yelling too much, no one knew it had exceeded and lot of energy and time was wasted with increasing errors in the syst…

One key insight of Shannon's is that to study information as he did, you need to remove "meaning". You could measure the information content of anything and it doesn't matter what the symbols mean. Paradoxically, a random sequence of characters of the same length as this message would contain more information than this message that I am writing.

---

The connection you draw to social media is cute, and I think it could help as an analogy in some cases. The Bit Player (2018 movie) makes a similar cute analogy to robust communication in marriage, that rather than saying the same message louder (yelling) or repeating the message (nagging), you should find many different ways to say the same message. And you could add an error-correcting code like "I love you".

I personally think it's a cute (not disparaging) way to think about communication among humans, but note that we still need to rely heavily on meaning, to decide what it even means to say the same message with different words.

What goes on with social media can be seen through lenses such as decision theory, and game theory, and graph theory, ..., and I think those fields have more to offer from the get-go than information theory. In any case, I personally would lean heavily on social science to try to get a better understanding of social situations, and to consider possible "interventions".

In particular, I do not think the kinds of communication errors that occur between humans are the same as e.g. bit flips. And to go back to the original point, it's tempting to conflate "information" (as in information theory) with "knowledge", or "understanding", or "truth".

Re: Claude Shannon: Tinkerer, Prankster, and Father of Information Theory (2016)

#28

Post by JoeDaDude »

Claude Shannon was also an early pioneer in artificial intelligence, the minimax algorithm applied to game trees was one of his inventions. He built several machines to play games and demonstrate his ideas, some of which can be seen in the article photos.

I compiled a list of his game gadgets, most of which still exist in the care of the MIT museum.

https://boardgamegeek.com/geeklist/143233/claude-shannon-man...

Re: Claude Shannon: Tinkerer, Prankster, and Father of Information Theory (2016)

#29

Post by cmehdy »

gugagore wrote: post #27
op03 wrote: post #3

From what I understand, before Shannon showed up no one knew how to calculate the Channel Capacity of a medium and how to deal with noise in the medium. So people just yelled louder or yelled repeatedly to get things across with better reliability. And when channel capacity was exceeded with everyone yelling too much, no one knew it had exceeded and lot of energy and time was wasted with increasing errors in the syst…

One key insight of Shannon's is that to study information as he did, you need to remove "meaning". You could measure the information content of anything and it doesn't matter what the symbols mean. Paradoxically, a random sequence of characters of the same length as this message would contain more information than this message that I am writing. --- The connection you draw to social media is cute, and I think it coul…

Now we need an analog of Shannon in social sciences to offer a Theory of Meaning. Then perhaps a century or so where there are still incompatibilities with the theory of information that prevent a bigger picture from emerging, and eventually an all-encompassing theory ushering humanity into an Era of Meaning.

Ted Chiang could probably write a plausible future with that as a writing prompt.

Re: Claude Shannon: Tinkerer, Prankster, and Father of Information Theory (2016)

#30

Post by idatum »

Shatnerz wrote: post #10

I recommend reading "A Mind at Play", a biography about Shannon. I found it to be a decent read. Funnily enough, I knew about him more through the unicycling community than I did through computer science. It wasn't until a few years ago when I realised what sort of impact he had on the world.

I also enjoyed "A Mind at Play".

I had forgotten (or never fully appreciated) what Shannon did and what other early pioneers did in computing. I have a better appreciation for all of them now.

One other highlight was Shannon's early childhood and using electrified fencing to communicate with others in his farmland neighborhood.

Post reply on HN