Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exploring Word2Vec

sujayskumar.blogspot.com

Post reply on HN
Print view View original

11–18 of 18 posts

Re: Exploring Word2Vec

#11

Post by yorwba »

Cybiote wrote: post #10
yorwba wrote: post #6

Earlier quoted context omitted.

The Gaussian mixture representation is quite interesting, so thanks for the link. But it seems like they do not make use of the relative position of words in the text either. They simply maximize the margin of the expected likelihood for words that occur in the same context over those that do not. However, in most cases it's probably actually better to ignore word order, since that captures more semantic relatedness…

Whether it's okay to ignore word order really depends on your goals. In my work, tracking word order to build a sentence representation is necessary to get composition rules correct. You can get away with ignoring order surprisingly often but when it's needed, it's vital. When aggregated across an entire corpus, order ends up mattering a lot, in an absolute sense (something looking only at accuracy numbers will hide)…

Right, what I meant was not that order doesn't matter at all, but that order mostly doesn't matter for the meaning of individual words (except for some homonyms which can be disambiguated using syntactical information).

When you want to handle whole sentences, obviously order becomes much more relevant, but for that you'd feed the word vectors into an LSTM or similar to handle the order-dependence.

Re: Exploring Word2Vec

#12

Post by sujayskumar »

stochastic_monk wrote: post #9

I enjoyed this writeup, as well as the canonical word2vec explanation linked. I will say that he’s wrong that NFL never co-occurs with ML. I’ve had discussions involving the No Free Lunch theorem by its initials.

Valid point. I should have made it clearer that NFL stands for National Football League. I was trying to demonstrate the difference in domain, one being sports and the other being computer science.

Re: Exploring Word2Vec

#13

Post by stochastic_monk »

sujayskumar wrote: post #12
stochastic_monk wrote: post #9

I enjoyed this writeup, as well as the canonical word2vec explanation linked. I will say that he’s wrong that NFL never co-occurs with ML. I’ve had discussions involving the No Free Lunch theorem by its initials.

Valid point. I should have made it clearer that NFL stands for National Football League. I was trying to demonstrate the difference in domain, one being sports and the other being computer science.

I understand, it’s just a funny coincidence.

Re: Exploring Word2Vec

#14

Post by neoncontrails »

swyx wrote: post #8
Jhsto wrote: post #7

Can someone explain why isn't the word 'the' considered part of the vocabulary in the blog post?

usual procedure in NLP: https://en.wikipedia.org/wiki/Stop_words

Word2Vec actually employs a default strategy of pruning the most common 0.0001% of words in the corpus. My understanding is that this permits the RNN layer to train on the unadulterated text rather than an unnatural concentrate of tokens, which is supposedly beneficial.

Interestingly there's some controversy whether this benefit extends to punctuation. Most W2V tutorials instruct the programmer to apply tokenizing functions that leave only space-separated word characters, annihilating everything else, but I've actually observed higher accuracy scores from models that tokenize non-word characters individually, i.e. as strings of length 1. I'm not sure exactly why this is the case, but there is a common sense interpretation which might explain. Output from the precursory tokenizer output is often transduced by word2phrase, an optional bigram recognizer. Mikolov strongly recommends using it, but I'm sure it struggles to infer pointwise mutual information from a text that's missing ~70% of its contents.

Re: Exploring Word2Vec

#15

Post by neoncontrails »

Cybiote wrote: post #10
yorwba wrote: post #6

Earlier quoted context omitted.

The Gaussian mixture representation is quite interesting, so thanks for the link. But it seems like they do not make use of the relative position of words in the text either. They simply maximize the margin of the expected likelihood for words that occur in the same context over those that do not. However, in most cases it's probably actually better to ignore word order, since that captures more semantic relatedness…

Whether it's okay to ignore word order really depends on your goals. In my work, tracking word order to build a sentence representation is necessary to get composition rules correct. You can get away with ignoring order surprisingly often but when it's needed, it's vital. When aggregated across an entire corpus, order ends up mattering a lot, in an absolute sense (something looking only at accuracy numbers will hide)…

This is much more nuanced criticism of language vectors than Chomsky's impatient hatred of them. There are interviews worth watching in which he loathes them for all the reasons you mention. Thankfully, I suppose, they rarely disappoint me. :)

Re: Exploring Word2Vec

#16

Post by neoncontrails »

patelajay285 wrote: post #2

We recently open sourced a library that makes it easy to quickly get started with pre-trained word vector models like word2vec from Google, Facebook, and Stanford. It adds a ton of extra functionality like fast similiarity indexing with Annoy and fairly robust out-of-vocabulary word lookups (handling misspellings) out of the box. If you want to quickly get started with pre-built models, it might be worth checking out…

Are document vectors supported analogously to word vectors? It's a bit peculiar to me that gensim withholds useful word2vec methods from the Doc2Vec class, like most_similar_from. I'd really enjoy having a common interface to both vector spaces.

Re: Exploring Word2Vec

#17

Post by patelajay285 »

neoncontrails wrote: post #16
patelajay285 wrote: post #2

We recently open sourced a library that makes it easy to quickly get started with pre-trained word vector models like word2vec from Google, Facebook, and Stanford. It adds a ton of extra functionality like fast similiarity indexing with Annoy and fairly robust out-of-vocabulary word lookups (handling misspellings) out of the box. If you want to quickly get started with pre-built models, it might be worth checking out…

Are document vectors supported analogously to word vectors? It's a bit peculiar to me that gensim withholds useful word2vec methods from the Doc2Vec class, like most_similar_from. I'd really enjoy having a common interface to both vector spaces.

Currently, it doesn't support document vectors, but it should be possible to add it and maintain the same interface. We're open to accepting a pull request if your interested in building that feature.

Re: Exploring Word2Vec

#18

Post by kamalbanga »

patelajay285 wrote: post #2

We recently open sourced a library that makes it easy to quickly get started with pre-trained word vector models like word2vec from Google, Facebook, and Stanford. It adds a ton of extra functionality like fast similiarity indexing with Annoy and fairly robust out-of-vocabulary word lookups (handling misspellings) out of the box. If you want to quickly get started with pre-built models, it might be worth checking out…

Hey, I like the work that you guys are doing and have been working on similar things since last 2 years. Are you guys hiring? My personal email id is banga.kamal@gmail.com.
Post reply on HN