Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rappers, Sorted by Size of Vocabulary

rappers.mdaniels.com.s3-website-us-east-1.amazonaws.com

Post reply on HN
Print view View original

221–230 of 276 posts

Re: Rappers, Sorted by Size of Vocabulary

#221

Post by danielsf »

logicallee wrote: post #108
simonster wrote: post #102

Earlier quoted context omitted.

Yes, I missed that, even though it is very clearly spelled out (oops!). It makes the ordinal comparison valid (modulo noise), but it does not completely address the concern. If you have two artists, and artist 1 uses 5,000 unique words in 35,000 lyrics while artist 2 uses 10,000 unique words in 35,000 lyrics, artist 2's vocabulary may be substantially more than twice as large as artist 1's. It is unlikely that a lyri…

This is a very good point - Aesop Rock, for example, uses one unique word every 5 words (7k unique in 35k), and if this does not stop, maybe he would continue and we would find the same average in 70k or 120k words. After all, you still have to have filler words like "to", "a", "the", "have", etc - he could be saturating the spots where he can put uniques. So this could substantially underrepresent vocabularies. Ther…

OP here. vocab is non-linear:

mdaniels.com/vocab/scatter.png

so the average would change as your sample size grows.

35K was the threshold where things didn't get garbled (like the 100 word example that you mention).

The threshold is also impacted by who I include. If I went to 50K, I'd lose out on rappers like Drake.

Re: Rappers, Sorted by Size of Vocabulary

#222

Post by danielsf »

Retric wrote: post #112
simonster wrote: post #102

Earlier quoted context omitted.

Yes, I missed that, even though it is very clearly spelled out (oops!). It makes the ordinal comparison valid (modulo noise), but it does not completely address the concern. If you have two artists, and artist 1 uses 5,000 unique words in 35,000 lyrics while artist 2 uses 10,000 unique words in 35,000 lyrics, artist 2's vocabulary may be substantially more than twice as large as artist 1's. It is unlikely that a lyri…

Repetition also seems like an huge issue given a 35,000 lyric limit. If someone repeats the same line 5-7 times it's hardly reasonable to count that when estimating vocabulary. Edit: A quick check found 7 repetitions of the same 8 word phrase in a DMX song. Which I chose becase he was at the bottom of the list. http://www.azlyrics.com/lyrics/dmx/whatthesebitcheswant.html

this is an issue that i was hoping would cancelled out, given the fact that I use the same analysis for every rapper. In short, it's an exact unique word count, but it's relationally accurate.

Re: Rappers, Sorted by Size of Vocabulary

#223

Post by danielsf »

benihana wrote: post #12

I'd really like to see this broken down by established vocabulary and made up vocabulary. I think that would really start to show who were the best lyricists on both ends. Rappers with a lot of made up words might be on the far left, and rappers with a lot of unique words that aren't made up would be on the far right. Both sides of the scale would show rapping talent on different dimensions. Influential rappers like…

OP here. made up vocabulary is basically all of hip hop. look at any lyrics file and very little is "established", yaknowwhatimsayin?

Re: Rappers, Sorted by Size of Vocabulary

#224

Post by danielsf »

benihana wrote: post #135
coherentpony wrote: post #66

Earlier quoted context omitted.

>Rappers with a lot of made up words might be on the far left It's interesting that you think that. I'd recommend you look at this: http://www.shakespeare-online.com/biography/wordsinvented.ht... At what point do 'made up' words become 'established'. After they've been published? If so, every made up word in a song should be considered 'established'.

I didn't mean to imply that being on the far left or being made up meant 'worse' in any way. Quite the opposite in fact - I think that if someone can create a word or phrase that becomes normal in everyday usages, it's a sign of... something. Maybe influence, maybe genius. Like you said, Shakespeare first used words that we consider common. I have no idea at what point a word becomes 'established' though.

I actually revised the conclusion to address this issue. Check it out on the site.

Re: Rappers, Sorted by Size of Vocabulary

#225

Post by danielsf »

loso wrote: post #158

I enjoyed reading this chart but I hope it doesn't reinforce the bias that some fans have that word complexity is the only way to tell if a rapper is good or not. There are several ways to judge the strength and weaknesses of a rapper. Complexity is one of them, flow is another. Story telling ability is also another very strong in indicator. The best rappers are able to bring a mix while some are just so strong in on…

I never made the claim that complexity is better/worse. I just wanted to communicate the data point, not imply 2pac>DMX.

Re: Rappers, Sorted by Size of Vocabulary

#226

Post by danielsf »

thrownaway2424 wrote: post #94

Gotta wonder about the garbage-in factor of Rap Genius. From one randomly selected Aesop Rock cut: "Please I want to donate my brain to the monstrous Panasonic profit" I guess it could be. I always heard it as "monstrous Panasonic prophet." It would be in keeping with the previous lyric "Television, all hail grand pixelated god of fantasy."

since i use only rap genius data, i'm hoping the garbage all cancels out so that the rappers are still relatively in the right place.

Re: Rappers, Sorted by Size of Vocabulary

#227

Post by danielsf »

unfunco wrote: post #14

This is fascinating. I'm only a recent listener of hip-hop (primarily because of Earl Sweatshirt and Odd Future) and I'm in awe of the vernacular. And similarly, as a boredom exercise a few weeks ago I did some lexical analysis of the song Timber (the monstrosity was being constantly played on the radio at the time) and here's what I came out with: "83.1% of the words in the lyrics are five letters or less, 58.9% are…

this is brilliant. OP here. We should work together.

Re: Rappers, Sorted by Size of Vocabulary

#228

Post by danielsf »

seizethecheese wrote: post #124

Many here seem to be interpreting vocabulary size as a signal for quality. When it comes to rap I completely disagree. Firstly, the repetition is rap's main ingredient. I read an article a while ago where researchers found that listening to a spoken phrase that is looped activates the same part of the brain as music, which helps explain this phenomenon. Personally, if I want food for thought I read. Rap is not an int…

I never made the claim that vocab size was correlated with quality.

Re: Rappers, Sorted by Size of Vocabulary

#229

Post by danielsf »

krick wrote: post #47

Really interesting, but not as representative as it should be. It's not clear why some have larger vocabulary than others. It could be using words like "zeitgeist" (in case of Aesop Rock) or some clever wordplay (I don't know much about hip-hop, so I can't find example for some artist from the list right off the bat, but I remember Marilyn Manson using word "gloominati" for instance) or pretty meaningless made up wor…

OP here. Do I really need to provide all of this to satisfy the reader's ability to grasp the basic premise of the site? this isn't a thesis or academic pursuit, just comparing some rappers for fun.

I used plain NLTK token analysis on rap genius lyrics. in terms of several more data slices...I agree that there should be more cuts of the data, but you must understand the amount of time that it took me to put this together.

Post reply on HN