Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vladimir Vapnik Joins Facebook Research

facebook.com

Post reply on HN
Print view View original

11–20 of 28 posts

Re: Vladimir Vapnik Joins Facebook Research

#11

Post by Smerity »

nl wrote: post #9
robrenaud wrote: post #4

Earlier quoted context omitted.

There is great, big data driven research coming out of Stanford using Common Crawl. For example, see http://www-nlp.stanford.edu/projects/glove/ . They successfully train an 840 billion token corpus. Vapnik is a big theory guy. Though I am not sure he has done anything of big practical importance recently, his immense contribution to ML (the SVM) was done at a time when machines were many orders of magnitudes weaker…

There is great, big data driven research coming out of Stanford using Common Crawl. For example, see http://www-nlp.stanford.edu/projects/glove/ . They successfully train an 840 billion token corpus. I haven't seen this paper before (thanks!!). How different is it to Word2Vec? Clearly the pre-trained vectors at that scale (and much bigger than the ones released with Word2Vec) are new and very exciting.

The paper compares in detail against word2vec, but (spoiler alert) GloVe using 42 billion tokens from Common Crawl beats word2vec using 100 billion tokens from the Google News corpus!

They don't actually use the 840 billion token model in the paper as it was made with some parameters that didn't allow for direct comparison, but the code and the models are all released for anyone to use from their site.

This is one of many great examples of open datasets like Common Crawl allowing talented people from academia and start-ups to compete with the large proprietary datasets of Google or Bing.

(disclaimer: data scientist at Common Crawl who does the crawling)

Re: Vladimir Vapnik Joins Facebook Research

#12

Post by nl »

Smerity wrote: post #11
nl wrote: post #9

Earlier quoted context omitted.

There is great, big data driven research coming out of Stanford using Common Crawl. For example, see http://www-nlp.stanford.edu/projects/glove/ . They successfully train an 840 billion token corpus. I haven't seen this paper before (thanks!!). How different is it to Word2Vec? Clearly the pre-trained vectors at that scale (and much bigger than the ones released with Word2Vec) are new and very exciting.

The paper compares in detail against word2vec, but (spoiler alert) GloVe using 42 billion tokens from Common Crawl beats word2vec using 100 billion tokens from the Google News corpus! They don't actually use the 840 billion token model in the paper as it was made with some parameters that didn't allow for direct comparison, but the code and the models are all released for anyone to use from their site. This is one of…

The paper compares in detail against word2vec, but (spoiler alert) GloVe using 42 billion tokens from Common Crawl beats word2vec using 100 billion tokens from the Google News corpus!

Damn!!

Background for those who don't follow this field: Word2Vec is an apparently miraculous demonstration and poster-child of the unreasonable effectiveness of big data. Beating it at all is impressive, assuming the performance is as robust as Word2Vec is against different metrics.

Beating it with only 42% of the tokens is wondrous.

Re: Vladimir Vapnik Joins Facebook Research

#13

Post by ajtulloch »

Smerity wrote: post #11
nl wrote: post #9

Earlier quoted context omitted.

There is great, big data driven research coming out of Stanford using Common Crawl. For example, see http://www-nlp.stanford.edu/projects/glove/ . They successfully train an 840 billion token corpus. I haven't seen this paper before (thanks!!). How different is it to Word2Vec? Clearly the pre-trained vectors at that scale (and much bigger than the ones released with Word2Vec) are new and very exciting.

The paper compares in detail against word2vec, but (spoiler alert) GloVe using 42 billion tokens from Common Crawl beats word2vec using 100 billion tokens from the Google News corpus! They don't actually use the 840 billion token model in the paper as it was made with some parameters that didn't allow for direct comparison, but the code and the models are all released for anyone to use from their site. This is one of…

The GloVe vs word2vec is not as clear-cut as that - see https://plus.google.com/114479713299850783539/posts/BYvhAbgG..., https://docs.google.com/document/d/1ydIujJ7ETSZ688RGfU5IMJJs... for more discussion.

Re: Vladimir Vapnik Joins Facebook Research

#14

Post by nl »

ajtulloch wrote: post #13
Smerity wrote: post #11

Earlier quoted context omitted.

The paper compares in detail against word2vec, but (spoiler alert) GloVe using 42 billion tokens from Common Crawl beats word2vec using 100 billion tokens from the Google News corpus! They don't actually use the 840 billion token model in the paper as it was made with some parameters that didn't allow for direct comparison, but the code and the models are all released for anyone to use from their site. This is one of…

The GloVe vs word2vec is not as clear-cut as that - see https://plus.google.com/114479713299850783539/posts/BYvhAbgG... , https://docs.google.com/document/d/1ydIujJ7ETSZ688RGfU5IMJJs... for more discussion.

Interesting.

As I said in my other comment, one of the strengths of Word2Vec is how robust it is against various metrics.

While it looks like GloVe's advantages over Word2Vec may be not as much as initially claimed, it is mostly as robust (which is good). However, the jump in Word+Context over just Word vectors when evaluated on semantic relations is interesting.

(To be clear: I'm very interested be being able to use the same system over diverse datasets, without having to tune it differently for each system - hence my interest in the robustness of the methodologies)

Edit: Were you and Smerity at Sydney Uni at the same time?

Re: Vladimir Vapnik Joins Facebook Research

#15

Post by Teodolfo »

iandanforth wrote: post #2

This is another great example of the unreasonable effectiveness of data. LeCunn, Hinton, Ng, Vapnik were all recruited on the basic fact that there is simply no way to do cutting edge research today without access to the data and computing resources of Google/Facebook/Yahoo/Baidu. Edit: "No way" is inaccurate. I should have said it is much easier to do at these companies. Also it is inaccurate to imply this is the on…

That is not accurate at all. These people can and did do lots of research in academia and had plenty of data. They obviously get paid a lot more in industry, however.

Re: Vladimir Vapnik Joins Facebook Research

#16

Post by j2kun »

iandanforth wrote: post #2

This is another great example of the unreasonable effectiveness of data. LeCunn, Hinton, Ng, Vapnik were all recruited on the basic fact that there is simply no way to do cutting edge research today without access to the data and computing resources of Google/Facebook/Yahoo/Baidu. Edit: "No way" is inaccurate. I should have said it is much easier to do at these companies. Also it is inaccurate to imply this is the on…

I think the real reason is because Facebook plans to recreate the Bell labs style of industry research, where researchers have license to do whatever they want.

Re: Vladimir Vapnik Joins Facebook Research

#17

Post by moab »

Teodolfo wrote: post #15
iandanforth wrote: post #2

This is another great example of the unreasonable effectiveness of data. LeCunn, Hinton, Ng, Vapnik were all recruited on the basic fact that there is simply no way to do cutting edge research today without access to the data and computing resources of Google/Facebook/Yahoo/Baidu. Edit: "No way" is inaccurate. I should have said it is much easier to do at these companies. Also it is inaccurate to imply this is the on…

That is not accurate at all. These people can and did do lots of research in academia and had plenty of data. They obviously get paid a lot more in industry, however.

Sources? I've seen first hand the limits of the 'data' we had at school. Papers constantly citing in their experiments: "the largest real world graphs we are aware of are on the order of 1B vertices" (a twitter graph, from something like 2011. The other highly cited one is the live-journal graph).

There's a massive dearth of data in academia. This is also why you see people like Kleinberg working directly with facebook on network research.

Re: Vladimir Vapnik Joins Facebook Research

#18

Post by tlarkworthy »

iandanforth wrote: post #2

This is another great example of the unreasonable effectiveness of data. LeCunn, Hinton, Ng, Vapnik were all recruited on the basic fact that there is simply no way to do cutting edge research today without access to the data and computing resources of Google/Facebook/Yahoo/Baidu. Edit: "No way" is inaccurate. I should have said it is much easier to do at these companies. Also it is inaccurate to imply this is the on…

Vapnik is philosophically not big data. SVMs are data efficient, at the cost of O(n^3) partitioning algorithms. His work has been more about maximizing the utility of the data you have.

Re: Vladimir Vapnik Joins Facebook Research

#19

Post by Smerity »

ajtulloch wrote: post #13
Smerity wrote: post #11

Earlier quoted context omitted.

The paper compares in detail against word2vec, but (spoiler alert) GloVe using 42 billion tokens from Common Crawl beats word2vec using 100 billion tokens from the Google News corpus! They don't actually use the 840 billion token model in the paper as it was made with some parameters that didn't allow for direct comparison, but the code and the models are all released for anyone to use from their site. This is one of…

The GloVe vs word2vec is not as clear-cut as that - see https://plus.google.com/114479713299850783539/posts/BYvhAbgG... , https://docs.google.com/document/d/1ydIujJ7ETSZ688RGfU5IMJJs... for more discussion.

Good link, thanks for pointing it out. Re: not clear cut, that's always the case to varying degrees :) To quote the author's in document response to the Google Doc you just linked to:

"Update by Richard Socher (Nov 2014): This document is outdated and its concerns have been addressed in the final version of the GloVe paper. Glove gets better performance on the same training data when actually run to convergence. See last section of Glove paper for details."

This is a good example of peer review in academia beyond just the paper review committee -- other researchers point out concerns or issues with methodology and they're addressed by the authors or other contributors. It's also great that the initial concerns could be properly tested thanks to the open source nature of both projects.

I will admit I didn't discuss the intricacies of the evaluation in my few paragraphs above, I was primarily speaking to the broader point that open data is helping academia compete with the goliaths of industrial research! =]

Re: Vladimir Vapnik Joins Facebook Research

#20

Post by patkai »

Am I the only one assuming that this many excellent scientists moving from academia is a loss for science in general? Will they really publish research in the same way they did before?

(Make no mistake, I can fully understand them, professors paid 80k per year, lacking resources, fighting bureaucrats, it is a great thing that they are recognised and at last paid what they deserve for devoting their lives to science.)

Post reply on HN