For a more thorough exploration of Markov text generation, I like this article: https://zompist.com/markov.html
Using a Markov chain to generate readable nonsense with 20 lines of Python
11–20 of 77 posts
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#12I wonder how this would behave if trained on 1 trillion words like the LLMs. Also, Is training a Markov cheaper that training neural nets? It would be a great way to cut AI costs if they could be made as effective as neural nets. There is also an interesting post by Orange duck. [0] [0] https://theorangeduck.com/page/17-line-markov-chain
The only way to make it sound less like a word salad is to increase the n-gram length (e.g. use the last 10 words to predict the next), but at that point it starts repeating the corpus.
I guess technically you can train on a huge corpus like those of NNs to mitigate that, but you’ll end up with more refined word salads then (edit: refined as in “locally coherent but globally still a word salad”)
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#13I wonder how this would behave if trained on 1 trillion words like the LLMs. Also, Is training a Markov cheaper that training neural nets? It would be a great way to cut AI costs if they could be made as effective as neural nets. There is also an interesting post by Orange duck. [0] [0] https://theorangeduck.com/page/17-line-markov-chain
Yeah, a century old approach using frequency tables where the next word is generated based just on the previous two words is highly competitive with contemporary neural nets. To produce coherent sentences, do you ever need to remember more than the last two words you said? I doubt it.
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#14We had a bot that would randomly say things in our IRC channel 15 years ago that worked like this. You could also mention it to prompt it to reply.
Every message was added to it's knowledge base and it would say random but hilarious stuff made up from all the nonsense we used to talk about.
Good times.
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#15[deleted]
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#16I was doing a similar experiment recently to generate random names that sound like names from a specific language.
I was breaking the list of names apart into 3 and 2 letter parts, marking which fragments are from the start, middle and end.
To generate the words I started from a random one from the start fragments, then continued with a random one from the middle fragments that starts with the latter that the previous one ended with, and similarly ended it with one from the end fragments. Some examples:
Spanish Names:
Armusa Vantara Modria
German Names:
Ven Marwar, Ger Naroff, Vort Kraldent, Görn Henter, Urg Wicher, Wan Ehranus, Eck Hayazin, Wert Biewin, Rein Relberid,
Catalan:
Pallava Ecorus Sangana Ginavari Telamita Exorxió
Hungarian cities:
Jószög Alszeny Hernafő Garnáza Ragytúr Hidácska Mezécs
(edit formatting)
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#17I wonder how this would behave if trained on 1 trillion words like the LLMs. Also, Is training a Markov cheaper that training neural nets? It would be a great way to cut AI costs if they could be made as effective as neural nets. There is also an interesting post by Orange duck. [0] [0] https://theorangeduck.com/page/17-line-markov-chain
The only way to make it sound less like a word salad is to increase the n-gram length (e.g. use the last 10 words to predict the next), but at that point it starts repeating the corpus. I guess technically you can train on a huge corpus like those of NNs to mitigate that, but you’ll end up with more refined word salads then (edit: refined as in “locally coherent but globally still a word salad”)
well - the lawyers from the NYT called, and they disagree...
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#18Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#19Earlier quoted context omitted.
The only way to make it sound less like a word salad is to increase the n-gram length (e.g. use the last 10 words to predict the next), but at that point it starts repeating the corpus. I guess technically you can train on a huge corpus like those of NNs to mitigate that, but you’ll end up with more refined word salads then (edit: refined as in “locally coherent but globally still a word salad”)
well - the lawyers from the NYT called, and they disagree...
I didn’t get the call, so it’s not clear what they disagree on :)
I doubt the have anything to say about Markov chains. We’re talking about technical possibilities, not legality of training corpora
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#20I did this once with Perl and a book called Eu by the Brazilian poet Augusto dos Anjos. It spat out verses and I joined a few to create this little thing:
Andam monstros sombrios pela escuridão dos remorsos
Pairando acima dos transeuntes
Maldito seja o gênero humano
Prostituído talvez em desintegrações maravilhosas
Source code: https://alquerubim.blogspot.com/2018/01/gerador-de-augusto-d...