How would it be possible that 40,000 words translates to 400,000 bits?! Am I missing something here?
Superposition / quantum mechanical
Researchers: It takes 1.5 MB of data to store language information
21–30 of 59 posts
Re: Researchers: It takes 1.5 MB of data to store language information
#22I haven't had a chance to look at it yet but here are the links to the paper and associated Open Science Framework data if anyone is interested:
Paper: https://royalsocietypublishing.org/doi/full/10.1098/rsos.181...
OSF Page: https://osf.io/ga9th/
Re: Researchers: It takes 1.5 MB of data to store language information
#23How would it be possible that 40,000 words translates to 400,000 bits?! Am I missing something here?
I've seen somewhere that there is about 1 bit of entropy per letter of English text. The best packer (Hutter prize) compresses 100MB of wikipedia down to 15MB, including the packer itself, a ratio of 1:6.5, which isn't that far off.
Considering that, 40k words to 400kbits is not too surprising.
Re: Researchers: It takes 1.5 MB of data to store language information
#24This describes how to summarize a language in a computer system. I think the storage requirements for the semantics is in particular questionable. If you can refer to a particular concept somewhere in the brain you only need a pointer to it. However, single pointers might not be sufficient The apparatus to learn a language might also not so easily distinguished from the structure to store the language or the structur…
>This describes how to summarize a language in a computer system.
It describes how to summarize a language from a information theory standpoint (which is something different).
There's a good reason to believe evolution would use something close to the most information theoretic efficient implementation (and even if not, it's a good lower bound)
Re: Researchers: It takes 1.5 MB of data to store language information
#25Earlier quoted context omitted.
Compression I suppose. Like storing the word "compression" is easier if you already know the familiar "com-" prefix as well as the noun "pression". Which is itself easier to remember if you know the verb "to press". I've been actively learning Portuguese and Russian lately, it's impressive how much faster I can pick up Portuguese vocabulary vs. Russian. And that's even for words that don't have an obvious cognate in…
"Compression" applies very much to grammatical variants of words. It's why the only bizarre irregular verbs in a language tend to be the ones that get used all the time - be, do, go etc - because for anything more obscure the brain just forgets the special case and applies the general rule. Steven Pinker's book Words and Rules is a great layman-oriented read if this sort of thing interests you.
It also makes sense why the most common verbs are irregular in most languages: to have us pick the direct word quickly, instead of the slower way of deriving it from a rule.
So, they're like "constants" vs calling a function to calculate a value.
Re: Researchers: It takes 1.5 MB of data to store language information
#26Of the various estimates that go into this number, I am most skeptical of the semantics one. I doubt that one can draw a clear line between our semantic appreciation of language and the totality of our remembered experiences.
Re: Researchers: It takes 1.5 MB of data to store language information
#27I haven't read the paper itself, but the article makes it sound like they simply counted the minimum number of bits required to represent the English language. It mentions no neuroscience-specific insights, so the "brain" part of the title is quite misleading.
Aside from that - is there really a way to express neural connections in the brain in terms of bits and bytes?
Re: Researchers: It takes 1.5 MB of data to store language information
#28How would it be possible that 40,000 words translates to 400,000 bits?! Am I missing something here?
The number of "bits" something takes is not an absolute value. It is relative to the encoding scheme that is being used for the bits.
Converting a large word list to a small number of bits has been a computer science hobby for a long time. Here's a pretty good search result to start working through for more details: https://duckduckgo.com/?q=building+a+small+spell+checker+suf... It was especially important to write small and fast spell checkers in the 1980s and early 1990s, when you couldn't expect to have enough RAM sitting around to simply load up a naively-encoded list of words, and the act of spell checking a few thousand words could take noticeable time.
So in an encoding scheme chosen to represent English compactly, I'm not too surprised that you can get things down quite small.
However, the question is, what relevance does that encoding scheme have to the human brain? Having just scanned through the paper, the answer is "probably not that much", which the researchers are well aware of. They explicitly present this as a lower-bound, which is a reasonable thing to do. It is obvious that the brain does not simply store 1.5MB of data in the way a computer does, in many ways.
To be honest, this amounts to an exercise in recreational mathematics more than anything else. There's nothing wrong with that, and that's not a criticism. My point is that I'm not sure it's worth trying to read the paper as anything else.
Re: Researchers: It takes 1.5 MB of data to store language information
#29I haven't read the paper itself, but the article makes it sound like they simply counted the minimum number of bits required to represent the English language. It mentions no neuroscience-specific insights, so the "brain" part of the title is quite misleading.
Aside from that - is there really a way to express neural connections in the brain in terms of bits and bytes?
The problem isn't whether there is "a" way; the problem is there's a multiplicity of ways and we don't know what the best one is. Part of the reason we don't have any idea what the best is is that we can't even construct any of them from a real brain, that is, the "brain scanning" technology the techno-rapture-style Singularity expects to develop does not even remotely exist right now. So we have nothing to experiment with or gain any experience with. The only ones we'd even remotely have access to right now would be some rather brute-force encoding of neurons and connectivity strengths and we pretty much know that such a thing must be incredibly redundant, but we have no ability right now to remove that redundancy. And we can't do it on a very large neural set.
It has been done on small models, though, at least to some significant extent: https://en.wikipedia.org/wiki/OpenWorm
Re: Researchers: It takes 1.5 MB of data to store language information
#30I haven't read the paper itself, but the article makes it sound like they simply counted the minimum number of bits required to represent the English language. It mentions no neuroscience-specific insights, so the "brain" part of the title is quite misleading.
Aside from that - is there really a way to express neural connections in the brain in terms of bits and bytes?
No. We still have a poor understanding of the exact way neurons code information and store information.
This page shows some of the current theories: https://en.wikipedia.org/wiki/Neural_coding#Hypothesized_cod...