Markov Chain Text Generation • Hacker News Board

Re: Markov Chain Text Generation

#11

Post by srean » Sun, May 12, 2019, 12:57 PM UTC

raverbashing wrote: post #8

xapata wrote: post #6

Earlier quoted context omitted.

In a sense, they're equivalent. RNN's create models of probabilistic state transitions. In other words, they create Markov chain models. You might argue that a Markov chain doesn't have memory, but it's easy to incorporate "memory" by expanding the definition of a "state" to include recent history or any function of the past.

No, they're not. And if it was "that easy" to incorporate memory RNNs would not have replaced Markov Chains As an example try to build a character level Markov chain for text generation and see how that goes.

> As an example try to build a character level Markov chain for text generation.

It does appear that you would benefit from such an excercise yourself. No really, just try it.

A well trained higher order hidden Markov model would function no less impressively than an RNN and with good reason.

Re: Markov Chain Text Generation

#12

Post by srean » Sun, May 12, 2019, 1:01 PM UTC

raverbashing wrote: post #10

aesthesia wrote: post #9

Earlier quoted context omitted.

The point is that an RNN or other language model describes a stochastic process which can be seen as a Markov process with state space given by the internal state of the RNN. This doesn't say anything about the ease of implementing or learning such a model. Just that from a purely mathematical perspective, they are equivalent in power.

Ah I see, I agree with their equivalence in the mathematical sense.

Am curious about what other sense of 'equivalence' you had in mind when you stated with great authority that RNNs are not Markovian.

Re: Markov Chain Text Generation

#13

Post by raverbashing » Sun, May 12, 2019, 1:58 PM UTC

srean wrote: post #12

raverbashing wrote: post #10

Earlier quoted context omitted.

Ah I see, I agree with their equivalence in the mathematical sense.

Am curious about what other sense of 'equivalence' you had in mind when you stated with great authority that RNNs are not Markovian.

In the practical sense, implementing a RNN is easier than a Markov Chain (libraries, etc)

Also RNNs can "evolve" (or be adapted) more easily to other architectures like LSTMs.

Re: Markov Chain Text Generation

#14

Post by raverbashing » Sun, May 12, 2019, 1:59 PM UTC

srean wrote: post #11

raverbashing wrote: post #8

Earlier quoted context omitted.

No, they're not. And if it was "that easy" to incorporate memory RNNs would not have replaced Markov Chains As an example try to build a character level Markov chain for text generation and see how that goes.

> As an example try to build a character level Markov chain for text generation. It does appear that you would benefit from such an excercise yourself. No really, just try it. A well trained higher order hidden Markov model would function no less impressively than an RNN and with good reason.

Looking at this, I'm almost convinced https://nbviewer.jupyter.org/gist/yoavg/d76121dfde2618422139 (but to me it still seems a LSTM can do a better job)

Re: Markov Chain Text Generation

#15

Post by srean » Sun, May 12, 2019, 2:37 PM UTC

raverbashing wrote: post #14

srean wrote: post #11

Earlier quoted context omitted.

> As an example try to build a character level Markov chain for text generation. It does appear that you would benefit from such an excercise yourself. No really, just try it. A well trained higher order hidden Markov model would function no less impressively than an RNN and with good reason.

Looking at this, I'm almost convinced https://nbviewer.jupyter.org/gist/yoavg/d76121dfde2618422139 (but to me it still seems a LSTM can do a better job)

Yup LSTMs can capture distant context, but they arent generic RNNs though. A significant part of "Deep" fanboyism comes from lack of knowledge about capabilities of conventional methods. I dont blame them entirely, there is a lot of diengenuous marketing that should claim partial credit.

Re: Markov Chain Text Generation

#16

Post by srean » Sun, May 12, 2019, 2:46 PM UTC

raverbashing wrote: post #13

srean wrote: post #12

Earlier quoted context omitted.

Am curious about what other sense of 'equivalence' you had in mind when you stated with great authority that RNNs are not Markovian.

In the practical sense, implementing a RNN is easier than a Markov Chain (libraries, etc) Also RNNs can "evolve" (or be adapted) more easily to other architectures like LSTMs.

That line of argument does not hold as RNNs are are just special cases of hidden Markov models. If RNNs are easy to implement it means some hidden Markov models are as well.

In any case by ease of implementation I think you men availability of libraries rather than something fundamental to the approach. Anyway we have strayed from your claim that RNNs are not Markovian. I was struck by the confidence in the claim especially so because it isnt true by a long shot.

Re: Markov Chain Text Generation

#17

Post by mratsim » Sun, May 12, 2019, 2:56 PM UTC

raverbashing wrote: post #13

srean wrote: post #12

Earlier quoted context omitted.

Am curious about what other sense of 'equivalence' you had in mind when you stated with great authority that RNNs are not Markovian.

In the practical sense, implementing a RNN is easier than a Markov Chain (libraries, etc) Also RNNs can "evolve" (or be adapted) more easily to other architectures like LSTMs.

If you try implementing a RNN from scratch with the same base as the OP N-order Markov Chain and without Numpy or a deep-learning frame work, I guarantee you that you will not find it so easy.

For reference his code his 400 lines and this is what he is importing:

    #include 
    #include 
    #include 
    #include 
    #include 
    #include 
    #include

We can assume that import is reasonable for comparison

To implement RNNs for text you need to implement tensors/ndarrays of 3 dimensions with slicint and all that jazz, efficient matrix multiplication, implement the RNN, implement the RNN backprop properly.

RNNs stands on the shoulder of giants.

Re: Markov Chain Text Generation

#18

Post by globuous » Sun, May 12, 2019, 4:03 PM UTC

monokai_nl wrote: post #3

Years ago I used Twitter streams as input for markov chain text generation. It’s interesting to recognize your own writing style in what is essentially gibberish: http://yes.thatcan.be/my/next/tweet

really good work !! I had a great laugh reading generated text from Trump and Obama ^^

Re: Markov Chain Text Generation

#19

Post by mnagel » Sun, May 12, 2019, 4:18 PM UTC

I did something simliar at university and (amongst other things) ran it over (simple) music scores in Lilypond format. The resulting texts could be converted to MIDI format and played by a synthesizer. It was quite interesting to listen to those. The code is still online at https://github.com/mnagel/markov

Re: Markov Chain Text Generation

#20

Post by herval » Sun, May 12, 2019, 6:38 PM UTC

I have a Twitter bot that reads text from random Project Gutenberg books and spits out book passages: https://twitter.com/markovian_lit

In general, it’s a bit all over the place and convoluted, but sometimes it generates true gems. People find it via hashtags, and some folks even interact before realizing it’s a silly bot.