51–55 of 55 posts
" I wrote a simple program who counted how many times did each word appear in the [ The Silmarillion ]. " Out of curiosity, where did the author get a copy of the text to analyze? "Turin" is the only name in The Silmarillion graph? Interesting.
I got them in txt files, online. I own the original books too. I would have typed them in if I had all the time of the world, of course.
Great post, I love the thinking out loud thought process portions. Some suggestions: try removing stopwords (the, and, etc.), it'll bring out more variation on the analysis. Particularly in the circular graphs. Try unique n-gram analysis, I suspect that will show something interesting. At the very least 2-gram (bigram/digram) analysis might show something cool. Some other interesting comparative measures, since the a…
And thanks for all the suggestions! I've made a note of them all. Good to hear from people who know more about the topic than me.
You need to remove the stop words before visualizing them. Words such as 'the' 'and' etc don't need to be in the analysis
(I'm the author)
Where's the sentence length statistics? Average word length? Commas per sentence? All are surely much more indicative of reading difficulty than that "originality index" he came up with. Why didn't the author try the most obvious test of reading difficulty of all: Flesch-Kincaid? [1] [1] http://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readabil...
Hi, author here. First, it's she, not he. Second, because I hadn't heard about that obvious test at all back then. I never pretended to do a superserious scientific analysis but rather answer to the questions that came to my mind, by using a computer to validate hypothesis. But many thanks for the pointers & suggestions, though! I was thinking about rebuilding this to make it realtime+interactive so that more than on…
You need to remove the stop words before visualizing them. Words such as 'the' 'and' etc don't need to be in the analysis
They are included because they were the reason people gave when asked why "The Silmarillion" was so unreadable. (I'm the author)
So, in order to test the hypothesis "Silmarillion is harder to read because it has lots of stop words", you need to calculate the relative frequencies of lots of other texts and see if there's something special about Silmarillion's top 10 versus all other's.
Surely, you have already done that using LOTR and The_Hobbit, but a much bigger sample is needed. At the very least, you may want to use 10-15 other works of fantasy from different authors, and that will be just like a back-of-the-envelop test to see if it is worth to pursue this experiment with a statistically significant sample.
[edit] 1. Provided it is sufficiently large.