Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

arxiv.org

Post reply on HN
Print view View original

11–20 of 49 posts

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#11

Post by mark_l_watson »

This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data.

BTW, I went to the North American Association of Computational Linguistics conference in April and it seemed like half the papers used LSTM.

Edit: the NAACL 2016 papers are here: http://aclweb.org/anthology/N/N16/

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#12

Post by forgotpwtomain »

mark_l_watson wrote: post #11

This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. BTW, I went to the North American Association of Computational Linguistics conference in April and i…

> This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data.

As someone outside of this field, it seems to me that this kind of result should have been very obviously foreseeable, hindsight bias and all that of course - but I would never have considered Markov processes to be an adequate predictability model for natural language. Though obviously the formalized results are important.

Could someone with more knowledge comment on what the current working assumptions were prior to this paper and what the consequences would be?

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#14

Post by thomasahle »

forgotpwtomain wrote: post #12
mark_l_watson wrote: post #11

This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. BTW, I went to the North American Association of Computational Linguistics conference in April and i…

> This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. As someone outside of this field, it seems to me that this kind of result should have been very ob…

> I would never have considered Markov processes to be an adequate predictability model for natural language.

Would you consider LSTM an adequate model?

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#15

Post by jakub_h »

mark_l_watson wrote: post #11

This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. BTW, I went to the North American Association of Computational Linguistics conference in April and i…

> This seems like a very important paper

> mark_l_watson

Hah, I'll take your word for it, then. :) Are there any recent comprehensive monographs you'd recommend for state-of-the-art NPL, for someone who has yet to enter the field?

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#17

Post by osipov »

mark_l_watson wrote: post #11

This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. BTW, I went to the North American Association of Computational Linguistics conference in April and i…

@mark_l_watson -- the paper also seems important in another respect -- in the conclusion the authors suggest abandoning loss functions as optimization objective functions in machine learning and replacing them with mutual information functions.

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#18

Post by BenoitP »

> [...] A Hidden Dimension in Natural Language

Mmmh

> [...] We show that in many data sequences — from texts in different languages to melodies and genomes

Hum, ehrm

> [...] natural languages are poorly approximated by Markov processes.

Alright, alright

> [...] This model class captures the essence of probabilistic context-free grammars

Ok, ok

> [...] and cosmological inflation

Wat.

Out of nowhere, Creation of the Univerve.

-------------

I'm always baffled by the ability to draw parallels. Did a colleague take at peek at the screen and said, hey I have the same equations?

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#19

Post by arnsholt »

jakub_h wrote: post #15
mark_l_watson wrote: post #11

This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. BTW, I went to the North American Association of Computational Linguistics conference in April and i…

> This seems like a very important paper > mark_l_watson Hah, I'll take your word for it, then. :) Are there any recent comprehensive monographs you'd recommend for state-of-the-art NPL, for someone who has yet to enter the field?

Monographs aren't really a thing in NLP, beyond theses. There are a couple of reference books most people lean on, though:

* Foundations of Statistical Natural Language Processing by Manning and Schütze

* Speech and Language Processing by Jurafsky and Martin (which is being revised for a third edition, which you can look at: https://web.stanford.edu/~jurafsky/slp3/ )

Beyond that, you're basically stuck reading the research literature. On the up-side, most of that literature is freely available from the ACL anthology at http://aclweb.org/anthology/

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#20

Post by TheOtherHobbes »

thomasahle wrote: post #14
forgotpwtomain wrote: post #12

Earlier quoted context omitted.

> This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. As someone outside of this field, it seems to me that this kind of result should have been very ob…

> I would never have considered Markov processes to be an adequate predictability model for natural language. Would you consider LSTM an adequate model?

Personally, no. I think all of these models are essentially trivial and a long way from genuine NLP.

That doesn't mean they're not useful in very narrow domains. But language is pretty much the definition of the ultimate wide domain, and trying to cover it with statistical correlations makes as much sense as word counting Shakespeare to try to generate some new plays.

Post reply on HN