For a corpus much more complete on the long tail, try google books ngram viewer : https://books.google.com/ngrams. That uses full data from google's book-scanning endeavors, millions of books compared to millions of words in the originally linked article.
The Long Tail of the English Language
11–20 of 55 posts
Re: The Long Tail of the English Language
#12I'm guessing "icing" is more popular in Canada.
Re: The Long Tail of the English Language
#13So I'm wondering, if you just learned the 200 most popular words, you might get pretty far in learning a new language, no?
200? no. Try 2,000. That should be representable of a barely usable level of the language. If you are what might be described as fluent, you're probably at >20,000. Take this headline:
JACUZZIS FOUND EFFECTIVE IN TREATING PHLEBITIS
"EFFECTIVE" and "IN" are the only two words found in the first 2,000 words sorted by frequency, although "FOUND" is close. So with only 200 words you'd understand:
WNPHMMVF SBHAQ RSSRPGVIR IN GERNGVAT CUYROVGVF
2,000 words would give you:
WNPHMMVF FOUND EFFECTIVE IN GERNGVAT CUYROVGVF
And some grammar knowledge would tell you
WNPHMMVF(N) FOUND EFFECTIVE IN GERNGVAT(V) CUYROVGVF(N)
Which is enough to know that you only really need to look up "TREATING" in the dictionary to understand the gist of the sentence But it'd hardly pass as a fluent understanding...
Re: The Long Tail of the English Language
#14So I'm wondering, if you just learned the 200 most popular words, you might get pretty far in learning a new language, no?
While the frequency of words drops off precipitously, there's also the fact that the set of all thoughts one might want to convey is incredibly vast, so within any given conversation there will probably be at least a few words which otherwise rarely appear.
Re: The Long Tail of the English Language
#15Very nice product! As a sidenote: anyway knows a service that provides human associations? Eg What do you associate with "Hacker" as a person "Computer", "Night", "Internet"
I work on ConceptNet, which does this.
Re: The Long Tail of the English Language
#16I wonder how other languages compare to English. I know English is far from pure. "The problem with defending the purity of the English language is that English is about as pure as a cribhouse whore. We don't just borrow words; on occasion, English has pursued other languages down alleyways to beat them unconscious and rifle their pockets for new vocabulary." -- James Nicoll
An interesting and large list of "English" words from other languages: https://en.wikipedia.org/wiki/Lists_of_English_words_by_coun...
Re: The Long Tail of the English Language
#17'I' doesn't seem to work. It is very common.
https://books.google.com/ngrams/graph?content=I%2C+you&year_...
Re: The Long Tail of the English Language
#18'I' doesn't seem to work. It is very common. https://books.google.com/ngrams/graph?content=I%2C+you&year_...
Try lower-casing it.
EDIT: This looks like a case of a common programming antipattern: you don't care about the casing for comparison purposes, so instead of implementing a case-insensitive compare, you downcase the strings and call it a day. But that's inherently a loss of data, and not having that data will eventually come back to bite you.
Re: The Long Tail of the English Language
#19If you have the right kind of friends you can play "Who knows the most obscure word?". Everyone picks a word from memory, check each word's position (We used to use google ngrams, but this would work well.), whoever gets the least common word wins that round, repeat until it's not fun anymore. That's how I learned defenestrate, obsequious, and a few other words.
Re: The Long Tail of the English Language
#20If you have the right kind of friends you can play "Who knows the most obscure word?". Everyone picks a word from memory, check each word's position (We used to use google ngrams, but this would work well.), whoever gets the least common word wins that round, repeat until it's not fun anymore. That's how I learned defenestrate, obsequious, and a few other words.
First word I tried wasn't even present in the list (equivocate). How do you score that?