Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LexVec, a word embedding model written in Go that outperforms word2vec

github.com

Post reply on HN
Print view View original

21–30 of 45 posts

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#21

Post by lukasb »

loudmax wrote: post #14

If anyone else is wondering what the heck "word embedding" means, it's a natural language processing technique. Here's a nice blog post about it: http://sebastianruder.com/word-embeddings-1/ It can process something like this: king - man + woman = queen Neat-o.

Post starts off kind of dense. Finally we get to the section "Word embedding models" and I say "ah ha! here we'll get a concise definition." Cut to ...

Naturally, every feed-forward neural network that takes words from a vocabulary as input and embeds them as vectors into a lower dimensional space, which it then fine-tunes through back-propagation, necessarily yields word embeddings as the weights of the first layer, which is usually referred to as Embedding Layer.

Naturally.

(Thanks, I believe that it is a great blog post, but I might look elsewhere for an intro ... :)

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#22

Post by PaulHoule »

From the viewpoint of commercial applications I find this profoundly depressing.

When the state of the art for accuracy is 0.6 on some task, you are going to always be a bridesmaid and never a bride, but hey, you can get bragging rights cause you did well on Kaggle.

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#24

Post by oelmekki »

lukasb wrote: post #21
loudmax wrote: post #14

If anyone else is wondering what the heck "word embedding" means, it's a natural language processing technique. Here's a nice blog post about it: http://sebastianruder.com/word-embeddings-1/ It can process something like this: king - man + woman = queen Neat-o.

Post starts off kind of dense. Finally we get to the section "Word embedding models" and I say "ah ha! here we'll get a concise definition." Cut to ... Naturally, every feed-forward neural network that takes words from a vocabulary as input and embeds them as vectors into a lower dimensional space, which it then fine-tunes through back-propagation, necessarily yields word embeddings as the weights of the first layer,…

Would love such kind of intro too, especially since I'm working on a product that could greatly benefit from NPL and neural networks.

Could a kind person provide a good reference for something we could learn from?

Or (as I fear), are we past this short time in the beginning of a technic/science (I think about computing, here) when you can learn without going through academic studies?

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#25

Post by computerex »

PaulHoule wrote: post #22

From the viewpoint of commercial applications I find this profoundly depressing. When the state of the art for accuracy is 0.6 on some task, you are going to always be a bridesmaid and never a bride, but hey, you can get bragging rights cause you did well on Kaggle.

That depends, to be honest. 60% accuracy, depending on the task, is far better than guessing at random. Secondly, depending on the task, human performance may not be that great either. Combined with controls, heuristics and validation, these "weak" models can still be of great use in commercial settings.

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#28

Post by argonaut »

rspeer wrote: post #4

As pre-built word vectors go, Conceptnet Numberbatch [1], introduced less flippantly as the ConceptNet Vector Ensemble [2], already outperforms this on all the measures evaluated in its paper: Rare Words, MEN-3000, and WordSim-353. This fact is hard to publicize because somehow the luminaries of the field decided that they didn't care about these evaluations anymore, back when RW performance was around 0.4. I have ha…

Not to be too harsh, but if all you did was ensemble several different embeddings using methods that were mostly from other papers, it would be pretty obvious you'd get state of the art performance. But that is not very interesting from a novel theoretical, scientific, or even engineering perspective. It is certainly useful to the community, just not academic. Most people would consider it obvious. I could achieve state of the art performance on ImageNet if I created a larger ensemble of Inception v3 networks than the ensemble the paper used and fiddled around with different data augmentation tricks they were too lazy to use, but that's not very interesting.

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#29

Post by plusepsilon »

rspeer wrote: post #4

As pre-built word vectors go, Conceptnet Numberbatch [1], introduced less flippantly as the ConceptNet Vector Ensemble [2], already outperforms this on all the measures evaluated in its paper: Rare Words, MEN-3000, and WordSim-353. This fact is hard to publicize because somehow the luminaries of the field decided that they didn't care about these evaluations anymore, back when RW performance was around 0.4. I have ha…

Cool stuff! It wasn't described in the paper (afaict), but did you have ConceptNet embeddings as well? I can certainly see a way of creating embeddings by using all the links as context. (e.g. http://conceptnet5.media.mit.edu/web/c/en/knowledge)

edit: actually looks like you "retrofit" existing word embeddings by re-weighing based on strengths of the links.

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#30

Post by knicholes »

oelmekki wrote: post #24
lukasb wrote: post #21

Earlier quoted context omitted.

Post starts off kind of dense. Finally we get to the section "Word embedding models" and I say "ah ha! here we'll get a concise definition." Cut to ... Naturally, every feed-forward neural network that takes words from a vocabulary as input and embeds them as vectors into a lower dimensional space, which it then fine-tunes through back-propagation, necessarily yields word embeddings as the weights of the first layer,…

Would love such kind of intro too, especially since I'm working on a product that could greatly benefit from NPL and neural networks. Could a kind person provide a good reference for something we could learn from? Or (as I fear), are we past this short time in the beginning of a technic/science (I think about computing, here) when you can learn without going through academic studies?

While dense and not necessarily for the mathematically faint of heart, I've learned a ton about NLP and ANNs through http://cs224d.stanford.edu/.
Post reply on HN