Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

arxiv.org

Post reply on HN
Print view View original

31–40 of 49 posts

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#31

Post by mark_l_watson »

nschucher wrote: post #28
mark_l_watson wrote: post #25

Earlier quoted context omitted.

I think you might be pleasantly surprised by recent results using DL and LSTM for building models of natural language. The next advancement I would like to see is handling anaphora resolution (resolving pronouns to previous noun phrases in text, resolving words like 'there' to a place mentioned elsewhere in text, etc.) Progress has been so rapid that I bet I don't have to wait long.

Have you seen the results from Dynamic Memory Networks? [0] The relevant example from the paper: I: Jane went to the hallway. I: Mary walked to the bathroom. I: Sandra went to the garden. I: Daniel went back to the garden. I: Sandra took the milk there. Q: Where is the milk? A: garden Obviously just a toy task, but as you said, progress is rapid! [0]: http://arxiv.org/abs/1506.07285

Thanks for the link!

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#32

Post by stelfer »

mark_l_watson wrote: post #11

This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. BTW, I went to the North American Association of Computational Linguistics conference in April and i…

The paper itself is a mess. It's very interesting but they aren't doing themselves any favors. They need to clean up the presentation and get rid of the distracting phrases like "fail epically".

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#33

Post by Gargoyle »

BenoitP wrote: post #18

> [...] A Hidden Dimension in Natural Language Mmmh > [...] We show that in many data sequences — from texts in different languages to melodies and genomes Hum, ehrm > [...] natural languages are poorly approximated by Markov processes. Alright, alright > [...] This model class captures the essence of probabilistic context-free grammars Ok, ok > [...] and cosmological inflation Wat. Out of nowhere, Creation of the Un…

Worth noting the authors, Henry Lin and Max Tegmark, are both astrophysicists. Among other things.

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#34

Post by jakub_h »

BenoitP wrote: post #18

> [...] A Hidden Dimension in Natural Language Mmmh > [...] We show that in many data sequences — from texts in different languages to melodies and genomes Hum, ehrm > [...] natural languages are poorly approximated by Markov processes. Alright, alright > [...] This model class captures the essence of probabilistic context-free grammars Ok, ok > [...] and cosmological inflation Wat. Out of nowhere, Creation of the Un…

> Did a colleague take at peek at the screen and said, hey I have the same equations?

This would an interesting thing to try - a computer system that would scan all the papers for math and find parallels. I think we already have something like term indexing for deductive systems?

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#35

Post by Gargoyle »

stelfer wrote: post #32
mark_l_watson wrote: post #11

This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. BTW, I went to the North American Association of Computational Linguistics conference in April and i…

The paper itself is a mess. It's very interesting but they aren't doing themselves any favors. They need to clean up the presentation and get rid of the distracting phrases like "fail epically".

Good ol HN comments never disappoint.

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#36

Post by kough »

jakub_h wrote: post #34
BenoitP wrote: post #18

> [...] A Hidden Dimension in Natural Language Mmmh > [...] We show that in many data sequences — from texts in different languages to melodies and genomes Hum, ehrm > [...] natural languages are poorly approximated by Markov processes. Alright, alright > [...] This model class captures the essence of probabilistic context-free grammars Ok, ok > [...] and cosmological inflation Wat. Out of nowhere, Creation of the Un…

> Did a colleague take at peek at the screen and said, hey I have the same equations? This would an interesting thing to try - a computer system that would scan all the papers for math and find parallels. I think we already have something like term indexing for deductive systems?

This is theoretically very possible, and I know that at least a few people (http://ccimi.maths.cam.ac.uk/projects/create-semantic-search...) are working on it.

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#37

Post by evanb »

BenoitP wrote: post #18

> [...] A Hidden Dimension in Natural Language Mmmh > [...] We show that in many data sequences — from texts in different languages to melodies and genomes Hum, ehrm > [...] natural languages are poorly approximated by Markov processes. Alright, alright > [...] This model class captures the essence of probabilistic context-free grammars Ok, ok > [...] and cosmological inflation Wat. Out of nowhere, Creation of the Un…

Max Tegmark is a cosmologist who writes ~5 serious papers and then one "out there" paper.

https://en.wikipedia.org/wiki/Max_Tegmark

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#38

Post by slashcom »

mark_l_watson wrote: post #11

This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. BTW, I went to the North American Association of Computational Linguistics conference in April and i…

Forgive me, but I'm less impressed by the paper. As far as I can tell, they've only really shown that (1) language is recursive, which we know already; (2) markov models cannot capture recursive languages, which we've known; and (3) RNNs can, which we've known. But so can PCFGs and many other formalisms from the past 25 years, which they ignore.

I did not read it very closely though.

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#39

Post by spdustin »

Gargoyle wrote: post #33
BenoitP wrote: post #18

> [...] A Hidden Dimension in Natural Language Mmmh > [...] We show that in many data sequences — from texts in different languages to melodies and genomes Hum, ehrm > [...] natural languages are poorly approximated by Markov processes. Alright, alright > [...] This model class captures the essence of probabilistic context-free grammars Ok, ok > [...] and cosmological inflation Wat. Out of nowhere, Creation of the Un…

Worth noting the authors, Henry Lin and Max Tegmark, are both astrophysicists. Among other things.

Not to nitpick, Max Tegmark is a cosmologist. I only recently learned the difference when I called a cosmologist friend an astrophysicist. Cosmology deals with the big stuff, almost philosophically, like: "where did the universe come from" and "what is the fate of the universe", while astrophysics deals with the nature of the things within, like: "how do stars form" and "what happens when black holes collide".

Max is deeply invested in modeling, analysis and prediction software, and I suspect did the bulk of the work in the paper.

Henry Lin is a student who is focused on astrophysics. He gave an interesting TED talk (http://www.ted.com/speakers/henry_lin) a few years back about studying distant galaxy clusters.

Henry is energetic and almost viscerally inspired by the beauty of science and mathematics, such a wonderful quality! His voice is definitely in the prose of the paper.

Re: Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

#40

Post by jakub_h »

kough wrote: post #36
jakub_h wrote: post #34

Earlier quoted context omitted.

> Did a colleague take at peek at the screen and said, hey I have the same equations? This would an interesting thing to try - a computer system that would scan all the papers for math and find parallels. I think we already have something like term indexing for deductive systems?

This is theoretically very possible, and I know that at least a few people ( http://ccimi.maths.cam.ac.uk/projects/create-semantic-search... ) are working on it.

Very yummy, thanks!
Post reply on HN