Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language

Post by evanb » Wed, Jul 13, 2016, 3:56 PM UTC

slashcom wrote: post #38

mark_l_watson wrote: post #11

This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. BTW, I went to the North American Association of Computational Linguistics conference in April and i…

Forgive me, but I'm less impressed by the paper. As far as I can tell, they've only really shown that (1) language is recursive, which we know already; (2) markov models cannot capture recursive languages, which we've known; and (3) RNNs can, which we've known. But so can PCFGs and many other formalisms from the past 25 years, which they ignore. I did not read it very closely though.

Post by eximius » Wed, Jul 13, 2016, 4:58 PM UTC

eximius wrote: post #7

mrcactu5 wrote: post #5

The Bach data consists of 5727 notes from Partita No. 2 [11], with all notes mapped into a 12-symbol alphabet consisting of the 12 half-tones {C, C#, D, D#, E, F, F#, G, G#, A, A#, B} with all timing, volume and octave information discarded. I was good until the last part.

Was this a sample of text generated by a Markov chain or their 'new' model?

Post by crypto5 » Wed, Jul 13, 2016, 7:31 PM UTC

mark_l_watson wrote: post #25

TheOtherHobbes wrote: post #20

Earlier quoted context omitted.

Personally, no. I think all of these models are essentially trivial and a long way from genuine NLP. That doesn't mean they're not useful in very narrow domains. But language is pretty much the definition of the ultimate wide domain, and trying to cover it with statistical correlations makes as much sense as word counting Shakespeare to try to generate some new plays.

I think you might be pleasantly surprised by recent results using DL and LSTM for building models of natural language. The next advancement I would like to see is handling anaphora resolution (resolving pronouns to previous noun phrases in text, resolving words like 'there' to a place mentioned elsewhere in text, etc.) Progress has been so rapid that I bet I don't have to wait long.

Post by schoen » Wed, Jul 13, 2016, 11:41 PM UTC

jakub_h wrote: post #34

BenoitP wrote: post #18

> [...] A Hidden Dimension in Natural Language Mmmh > [...] We show that in many data sequences — from texts in different languages to melodies and genomes Hum, ehrm > [...] natural languages are poorly approximated by Markov processes. Alright, alright > [...] This model class captures the essence of probabilistic context-free grammars Ok, ok > [...] and cosmological inflation Wat. Out of nowhere, Creation of the Un…

> Did a colleague take at peek at the screen and said, hey I have the same equations? This would an interesting thing to try - a computer system that would scan all the papers for math and find parallels. I think we already have something like term indexing for deductive systems?