Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LexVec, a word embedding model written in Go that outperforms word2vec

github.com

Post reply on HN
Print view View original

31–40 of 45 posts

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#31

Post by aweinstock »

oelmekki wrote: post #24
lukasb wrote: post #21

Earlier quoted context omitted.

Post starts off kind of dense. Finally we get to the section "Word embedding models" and I say "ah ha! here we'll get a concise definition." Cut to ... Naturally, every feed-forward neural network that takes words from a vocabulary as input and embeds them as vectors into a lower dimensional space, which it then fine-tunes through back-propagation, necessarily yields word embeddings as the weights of the first layer,…

Would love such kind of intro too, especially since I'm working on a product that could greatly benefit from NPL and neural networks. Could a kind person provide a good reference for something we could learn from? Or (as I fear), are we past this short time in the beginning of a technic/science (I think about computing, here) when you can learn without going through academic studies?

This is fairly readable (high-level) post on word embeddings: http://colah.github.io/posts/2014-07-NLP-RNNs-Representation...

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#32

Post by rspeer »

argonaut wrote: post #28
rspeer wrote: post #4

As pre-built word vectors go, Conceptnet Numberbatch [1], introduced less flippantly as the ConceptNet Vector Ensemble [2], already outperforms this on all the measures evaluated in its paper: Rare Words, MEN-3000, and WordSim-353. This fact is hard to publicize because somehow the luminaries of the field decided that they didn't care about these evaluations anymore, back when RW performance was around 0.4. I have ha…

Not to be too harsh, but if all you did was ensemble several different embeddings using methods that were mostly from other papers, it would be pretty obvious you'd get state of the art performance. But that is not very interesting from a novel theoretical, scientific, or even engineering perspective. It is certainly useful to the community, just not academic . Most people would consider it obvious. I could achieve s…

Your claim of "I could do that" rings false.

You didn't do it. You think that you could have done it because I wrote a clear enough paper telling you how. And even if you were curious and determined enough to do it, the thing that would make it possible for you to do it are the years that I have spent making ConceptNet.

And when the "academic" things consistently do 10% or more worse than the "obvious" thing, maybe it's not that obvious.

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#33

Post by argonaut »

rspeer wrote: post #32
argonaut wrote: post #28

Earlier quoted context omitted.

Not to be too harsh, but if all you did was ensemble several different embeddings using methods that were mostly from other papers, it would be pretty obvious you'd get state of the art performance. But that is not very interesting from a novel theoretical, scientific, or even engineering perspective. It is certainly useful to the community, just not academic . Most people would consider it obvious. I could achieve s…

Your claim of "I could do that" rings false. You didn't do it. You think that you could have done it because I wrote a clear enough paper telling you how. And even if you were curious and determined enough to do it, the thing that would make it possible for you to do it are the years that I have spent making ConceptNet. And when the "academic" things consistently do 10% or more worse than the "obvious" thing, maybe i…

I was referring to ImageNet as to what I could do (because my experience is in computer vision). But more broadly, ensembling in general is really really obvious. I mean, for heavens sake, it's the trick everyone uses on Kaggle.

You will find that is a very common reason why academics dismiss work for being too incremental - it is too obvious. It doesn't matter how many years you spent working on it.

Whether or not your work is novel depends on what you did, not strictly on whether you managed to squeeze out more performance on a benchmark.

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#34

Post by rspeer »

plusepsilon wrote: post #29
rspeer wrote: post #4

As pre-built word vectors go, Conceptnet Numberbatch [1], introduced less flippantly as the ConceptNet Vector Ensemble [2], already outperforms this on all the measures evaluated in its paper: Rare Words, MEN-3000, and WordSim-353. This fact is hard to publicize because somehow the luminaries of the field decided that they didn't care about these evaluations anymore, back when RW performance was around 0.4. I have ha…

Cool stuff! It wasn't described in the paper (afaict), but did you have ConceptNet embeddings as well? I can certainly see a way of creating embeddings by using all the links as context. (e.g. http://conceptnet5.media.mit.edu/web/c/en/knowledge ) edit: actually looks like you "retrofit" existing word embeddings by re-weighing based on strengths of the links.

There kind of are ConceptNet embeddings. They weren't called "embeddings" when they were released, because it was 2008 [1]. Here's an example: http://conceptnet5.media.mit.edu/data/5.4/assoc/c/en/knowled...

We dropped working on those embeddings for a while because retrofitting was more effective. But we may have found a way to use the ConceptNet embeddings again, to improve languages besides English beyond what retrofitting is doing.

[1] http://dspace.mit.edu/handle/1721.1/51870

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#35

Post by rspeer »

argonaut wrote: post #33
rspeer wrote: post #32

Earlier quoted context omitted.

Your claim of "I could do that" rings false. You didn't do it. You think that you could have done it because I wrote a clear enough paper telling you how. And even if you were curious and determined enough to do it, the thing that would make it possible for you to do it are the years that I have spent making ConceptNet. And when the "academic" things consistently do 10% or more worse than the "obvious" thing, maybe i…

I was referring to ImageNet as to what I could do (because my experience is in computer vision). But more broadly, ensembling in general is really really obvious . I mean, for heavens sake, it's the trick everyone uses on Kaggle. You will find that is a very common reason why academics dismiss work for being too incremental - it is too obvious. It doesn't matter how many years you spent working on it. Whether or not…

Ensemble methods have been around for decades because they are a good idea. Watson -- the Jeopardy bot, not the marketing brand -- was an ensemble method.

My impression is that people avoid ensembles because they are fighting to give the impression that their work is the only work that matters. It's "not invented here" syndrome. I'm glad I'm not trying to get tenure.

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#36

Post by argonaut »

rspeer wrote: post #35
argonaut wrote: post #33

Earlier quoted context omitted.

I was referring to ImageNet as to what I could do (because my experience is in computer vision). But more broadly, ensembling in general is really really obvious . I mean, for heavens sake, it's the trick everyone uses on Kaggle. You will find that is a very common reason why academics dismiss work for being too incremental - it is too obvious. It doesn't matter how many years you spent working on it. Whether or not…

Ensemble methods have been around for decades because they are a good idea. Watson -- the Jeopardy bot, not the marketing brand -- was an ensemble method. My impression is that people avoid ensembles because they are fighting to give the impression that their work is the only work that matters. It's "not invented here" syndrome. I'm glad I'm not trying to get tenure.

People avoid ensembles, unless they're trying to squeeze out higher performance on a benchmark (or they're doing research on the technique of ensembling in of itself). I would have thought this would be obvious to you, since you're a researcher.

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#37

Post by rspeer »

argonaut wrote: post #36
rspeer wrote: post #35

Earlier quoted context omitted.

Ensemble methods have been around for decades because they are a good idea. Watson -- the Jeopardy bot, not the marketing brand -- was an ensemble method. My impression is that people avoid ensembles because they are fighting to give the impression that their work is the only work that matters. It's "not invented here" syndrome. I'm glad I'm not trying to get tenure.

People avoid ensembles, unless they're trying to squeeze out higher performance on a benchmark (or they're doing research on the technique of ensembling in of itself). I would have thought this would be obvious to you, since you're a researcher.

My company, Luminoso, uses Conceptnet Numberbatch as one component for interpreting the meaning of text. When the benchmarks went up, the understandability of its results did too. You get better search results, better topics, clearer visualizations.

I'm not just trying to squeeze out extra performance, I'm trying to make computers understand text better. The benchmarks are the evidence that it's better. I do consider myself a researcher despite leaving academia, and having some respect for evidence is part of that identity.

When academia decides to disregard evidence because evidence is for stupid Kagglers (I don't use Kaggle but I respect a good result when I see one), that's how they end up lagging behind open source software.

I understand that it's not worthwhile to chase every evaluation. For example, some evaluations are unrepresentative. Some evaluations, like parsing according to the Penn Treebank, were once useful but have been squeezed dry in a way that doesn't reflect real-world performance. And some tasks chase these unhelpful evaluations.

But I would credit Kaggle with making academics realize, slowly, that they should use random forests as a baseline when evaluating a classification method. People were content to publish classifiers that were worse than random forests until Kaggle presented overwhelming evidence that random forests worked better than many techniques.

In text understanding, the fact that seems not to have taken hold -- one that I think should be obvious, even -- is "computers understand text better when they know more facts about words". This is what ConceptNet (not the whole ensemble, but ConceptNet itself) is about.

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#38

Post by iraphael »

aweinstock wrote: post #31
oelmekki wrote: post #24

Earlier quoted context omitted.

Would love such kind of intro too, especially since I'm working on a product that could greatly benefit from NPL and neural networks. Could a kind person provide a good reference for something we could learn from? Or (as I fear), are we past this short time in the beginning of a technic/science (I think about computing, here) when you can learn without going through academic studies?

This is fairly readable (high-level) post on word embeddings: http://colah.github.io/posts/2014-07-NLP-RNNs-Representation...

And this is one that explains everything OP needs to understand the confusing sentence from their intro:

http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip...

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#39

Post by lmmlzxx »

IshKebab wrote: post #7

Has anyone done any work on handing words that have overloading meanings? Something like 'lead' has two really distinct uses. It's really multiple words that happened to be spelt the same.

Actually "lead" has many, many more than two distinct senses.

https://en.wiktionary.org/wiki/lead

Re: LexVec, a word embedding model written in Go that outperforms word2vec

#40

Post by gtani »

rpedela wrote: post #27

Slightly off-topic, but I thought this would be a good place to ask. Are there any word embedding tools which take a Lucene/Solr/ES index as input and output a synonyms file which can be used to improve search recall?

There's a few projects that use ES/lucene as a backend/datastore once the feature engineering is done, but I don't see models operating on the native indexes directly, maybe the format is too different from one-hot (after turning off stemming/stopwords and other info-losing steps)

http://lucene.472066.n3.nabble.com/Where-Search-Meets-Machin...

https://news.ycombinator.com/item?id=11876542

Post reply on HN