Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

arxiv.org

Post reply on HN
Print view View original

21–30 of 49 posts

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#21

Post by forgotpwtomain »

thomasahle wrote: post #14
forgotpwtomain wrote: post #12

Earlier quoted context omitted.

> This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. As someone outside of this field, it seems to me that this kind of result should have been very ob…

> I would never have considered Markov processes to be an adequate predictability model for natural language. Would you consider LSTM an adequate model?

> Deep models are important because without the extra “dimension” of depth/abstraction, there is no way to construct “shortcuts” between random variables that are separated by large amounts of time with short-range interactions; 1D models will be doomed to exponential decay.

From my non-professional perspective the above seems that it should have been very obvious - (and also that correlations between variables for natural languages would be better explained by multi-dimensional structure). That is if you told me that this were proved / formally supported as it is in this paper, my reaction would be a "that sounds like reasonable approximation" not - "that result sounds very surprising I must read the paper".

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#22

Post by mark_l_watson »

jakub_h wrote: post #15
mark_l_watson wrote: post #11

This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. BTW, I went to the North American Association of Computational Linguistics conference in April and i…

> This seems like a very important paper > mark_l_watson Hah, I'll take your word for it, then. :) Are there any recent comprehensive monographs you'd recommend for state-of-the-art NPL, for someone who has yet to enter the field?

If you have the time, I would start by taking Andrew Ng's machine learning class and then this NLP class when it is next offered in September 2016 https://www.coursera.org/learn/nlp

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#23

Post by jakub_h »

forgotpwtomain wrote: post #12
mark_l_watson wrote: post #11

This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. BTW, I went to the North American Association of Computational Linguistics conference in April and i…

> This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. As someone outside of this field, it seems to me that this kind of result should have been very ob…

> but I would never have considered Markov processes to be an adequate predictability model for natural language

Isn't it the case that for very short distances (several elements), power decay and exponential decay are (or can be made, with proper constants) more similar? Thus, if predictive models were originally studied only for very short sequences in the past (limited computational resources!), it seems to make sense that this is a mistake that anyone could have made more easily back then.

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#24

Post by jakub_h »

arnsholt wrote: post #19
jakub_h wrote: post #15

Earlier quoted context omitted.

> This seems like a very important paper > mark_l_watson Hah, I'll take your word for it, then. :) Are there any recent comprehensive monographs you'd recommend for state-of-the-art NPL, for someone who has yet to enter the field?

Monographs aren't really a thing in NLP, beyond theses. There are a couple of reference books most people lean on, though: * Foundations of Statistical Natural Language Processing by Manning and Schütze * Speech and Language Processing by Jurafsky and Martin (which is being revised for a third edition, which you can look at: https://web.stanford.edu/~jurafsky/slp3/ ) Beyond that, you're basically stuck reading the re…

Thank you very much. I'm aware of a large number of books that I could read, but if Sturgeon's law applies, or at least if a lot of the reading would be redundant anyway, I'll rather ask for pointers than to waste a lot of time by rediscovering the wheel. Basically, what I was asking for was the NLP's respected equivalent of Norvig and Russell, or similar books from other fields. Those two books seem to fit to bill.

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#25

Post by mark_l_watson »

TheOtherHobbes wrote: post #20
thomasahle wrote: post #14

Earlier quoted context omitted.

> I would never have considered Markov processes to be an adequate predictability model for natural language. Would you consider LSTM an adequate model?

Personally, no. I think all of these models are essentially trivial and a long way from genuine NLP. That doesn't mean they're not useful in very narrow domains. But language is pretty much the definition of the ultimate wide domain, and trying to cover it with statistical correlations makes as much sense as word counting Shakespeare to try to generate some new plays.

I think you might be pleasantly surprised by recent results using DL and LSTM for building models of natural language. The next advancement I would like to see is handling anaphora resolution (resolving pronouns to previous noun phrases in text, resolving words like 'there' to a place mentioned elsewhere in text, etc.) Progress has been so rapid that I bet I don't have to wait long.

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#26

Post by jakub_h »

hacker43 wrote: post #2

hmm...maybe the way we then encode data digitally is also "wrong" or should I say unnatural.

It seems to me that if we humans are the only ones doing this, it is either trivially natural or trivially unnatural based on whether you include or exclude our creations in/out of the natural world.

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#27

Post by ASpring »

BenoitP wrote: post #18

> [...] A Hidden Dimension in Natural Language Mmmh > [...] We show that in many data sequences — from texts in different languages to melodies and genomes Hum, ehrm > [...] natural languages are poorly approximated by Markov processes. Alright, alright > [...] This model class captures the essence of probabilistic context-free grammars Ok, ok > [...] and cosmological inflation Wat. Out of nowhere, Creation of the Un…

One of the common threads I've noticed between the best researchers I know is their uncanny ability to draw these parallels between the most obscure domains.

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#28

Post by nschucher »

mark_l_watson wrote: post #25
TheOtherHobbes wrote: post #20

Earlier quoted context omitted.

Personally, no. I think all of these models are essentially trivial and a long way from genuine NLP. That doesn't mean they're not useful in very narrow domains. But language is pretty much the definition of the ultimate wide domain, and trying to cover it with statistical correlations makes as much sense as word counting Shakespeare to try to generate some new plays.

I think you might be pleasantly surprised by recent results using DL and LSTM for building models of natural language. The next advancement I would like to see is handling anaphora resolution (resolving pronouns to previous noun phrases in text, resolving words like 'there' to a place mentioned elsewhere in text, etc.) Progress has been so rapid that I bet I don't have to wait long.

Have you seen the results from Dynamic Memory Networks? [0]

The relevant example from the paper:

  I: Jane went to the hallway.
  I: Mary walked to the bathroom.
  I: Sandra went to the garden.
  I: Daniel went back to the garden.
  I: Sandra took the milk there.
  Q: Where is the milk?
  A: garden
Obviously just a toy task, but as you said, progress is rapid!

[0]: http://arxiv.org/abs/1506.07285

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#29

Post by jakub_h »

mark_l_watson wrote: post #22
jakub_h wrote: post #15

Earlier quoted context omitted.

> This seems like a very important paper > mark_l_watson Hah, I'll take your word for it, then. :) Are there any recent comprehensive monographs you'd recommend for state-of-the-art NPL, for someone who has yet to enter the field?

If you have the time, I would start by taking Andrew Ng's machine learning class and then this NLP class when it is next offered in September 2016 https://www.coursera.org/learn/nlp

> Andrew Ng's machine learning class

Ah yes, the one I've heard about but still have to take. :) Well, I guess I should give Coursesa a chance. (Somehow I'm not fond of their "timelined" format, it seems redundant if you're communicating with a machine. I hope the future of online learning will avoid it like the plague.)

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#30

Post by jakub_h »

ASpring wrote: post #27
BenoitP wrote: post #18

> [...] A Hidden Dimension in Natural Language Mmmh > [...] We show that in many data sequences — from texts in different languages to melodies and genomes Hum, ehrm > [...] natural languages are poorly approximated by Markov processes. Alright, alright > [...] This model class captures the essence of probabilistic context-free grammars Ok, ok > [...] and cosmological inflation Wat. Out of nowhere, Creation of the Un…

One of the common threads I've noticed between the best researchers I know is their uncanny ability to draw these parallels between the most obscure domains.

Heh! For some reason, that reminded me of this famous conversation:

"...and that, my liege, is how we know the Earth to be banana-shaped."

"This new learning amazes me, Sir Bedevere. Explain again how sheep's bladders may be employed to prevent earthquakes."

Post reply on HN