Why literature is the ultimate big-data challenge
economist.com
Why literature is the ultimate big-data challenge
1–10 of 22 posts
Re: Why literature is the ultimate big-data challenge
#2I'm interested in this subject, as I've studied it before. Was really nice to see how many techniques are used, with those functional words being just one (with good results, it must be said). Even reducing words to n-grams is interesting, since it catches the radix of the words (leaving those prefixes and suffixes out of the question).
Re: Why literature is the ultimate big-data challenge
#3I would be interested in reading some technical papers on the subject, could someone post a few? Whether about Shakespeare or other literature analysis?
That said, this is SO not big data. It is small data, but potentially interesting analysis.
Re: Why literature is the ultimate big-data challenge
#4Although the field had its crackpots it was never as off-the-rails as the Economist writer makes it out to be. Mosteller and Wallace's analysis of the Federalist Papers is a well-known early effort (https://priceonomics.com/how-statistics-solved-a-175-year-ol...).
Part of my MA covered this topic (stylometry), you can take a look (https://www.dropbox.com/s/3q9ljrgnntgs6ee/t.pdf). It lists some of the references up until that time (c. 2004).
Re: Why literature is the ultimate big-data challenge
#5I would be interested in reading some technical papers on the subject, could someone post a few? Whether about Shakespeare or other literature analysis? That said, this is SO not big data. It is small data, but potentially interesting analysis.
I think it depends on what you mean. I don't think that just the number of bytes of data should be used to measure the size of your dataset. For example, if I have 1TB of all 1's, this is a lot of data, but not very interesting.
I think a more nuanced notion of size is the /information content/ of the datasets. I haven't thought about it carefully, but I'm sure you can quantify this more explicitly in terms of information theory or other non-bit-based complexity measures.
From this point of view, literature may not be large (in terms of the number of bytes), but in terms of information content, it is incredibly dense. A large portion of human knowledge is written somewhere. From this point of view, it is in fact big data.
Re: Why literature is the ultimate big-data challenge
#6Can't really agree, literature is literature because of form not content, every literary work is its own separate world with references to other similarly separate worlds from the past. Data analysis can help finding invariants, like in Meter in Poetry from Prof Nigel Fabb https://www.amazon.co.uk/Meter-Poetry-Theory-Nigel-Fabb/dp/0... , but not the reason why literature is literary at all, which comes from social sciences. Mythopoesis as the cumulative sum of the real and imaginary worlds thought by humans is another possible result but it would not be literature, languages and theory of language instead.
Re: Why literature is the ultimate big-data challenge
#7I would be interested in reading some technical papers on the subject, could someone post a few? Whether about Shakespeare or other literature analysis? That said, this is SO not big data. It is small data, but potentially interesting analysis.
Re: Why literature is the ultimate big-data challenge
#8Although the field had its crackpots it was never as off-the-rails as the Economist writer makes it out to be. Mosteller and Wallace's analysis of the Federalist Papers is a well-known early effort ( https://priceonomics.com/how-statistics-solved-a-175-year-ol... ). Part of my MA covered this topic (stylometry), you can take a look ( https://www.dropbox.com/s/3q9ljrgnntgs6ee/t.pdf ). It lists some of the references u…
Thanks!
Re: Why literature is the ultimate big-data challenge
#9I would be interested in reading some technical papers on the subject, could someone post a few? Whether about Shakespeare or other literature analysis? That said, this is SO not big data. It is small data, but potentially interesting analysis.
I think it depends on what you mean. I don't think that just the number of bytes of data should be used to measure the size of your dataset. For example, if I have 1TB of all 1's, this is a lot of data, but not very interesting. I think a more nuanced notion of size is the /information content/ of the datasets. I haven't thought about it carefully, but I'm sure you can quantify this more explicitly in terms of inform…
Sounds like you may be interested in https://en.m.wikipedia.org/wiki/Kolmogorov_complexity
Re: Why literature is the ultimate big-data challenge
#10I would be interested in reading some technical papers on the subject, could someone post a few? Whether about Shakespeare or other literature analysis? That said, this is SO not big data. It is small data, but potentially interesting analysis.
I think it depends on what you mean. I don't think that just the number of bytes of data should be used to measure the size of your dataset. For example, if I have 1TB of all 1's, this is a lot of data, but not very interesting. I think a more nuanced notion of size is the /information content/ of the datasets. I haven't thought about it carefully, but I'm sure you can quantify this more explicitly in terms of inform…
Maybe you would find the theory of information interesting:
http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...
Here in his famous tour de force Claude Shannon lays out the way we can estimate the amount of actual information in an act of communication (e.g. a literary work) and relates it to system entropy.
To your point, !TB of all 1's compresses to a just few bits of actual information.
But I suspect you are not actually speaking about the information contained in a literary work. You are probably thinking: how much expansive commentary and explanation could a literary work spawn? That is another question. The answer is always: unbounded.