Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using a Markov chain to generate readable nonsense with 20 lines of Python

benhoyt.com

Post reply on HN
Print view View original

31–40 of 77 posts

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#31

Post by bhattid »

alejoar wrote: post #14

We had a bot that would randomly say things in our IRC channel 15 years ago that worked like this. You could also mention it to prompt it to reply. Every message was added to it's knowledge base and it would say random but hilarious stuff made up from all the nonsense we used to talk about. Good times.

Out of curiosity, was the bot based on the bMotion repository? https://github.com/jamesoff/bmotion

I remember a friend of mine settings up an IRC bot (named Zeta) like that for his sheet music forum many years ago. She was involved in a lot of hilarity - probably my favorite antics were when she randomly decided to courtmatial someone. Good times indeed! :)

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#32

Post by 3abiton »

abetusk wrote: post #9

As many know and point out, this idea is now very old (40+ years?). The big problem is that it creates "word salad" [0]. In the generative art and procgen community, the term "10,000 bowls of oatmeal" problem has been used [1]. Taking a step back, this is a perennial problem, even with AI old and new. I've heard that the early (good) chess bots, after Deep Blue, had a problem of only being locally context sensitive a…

Any suggestiob on the background of the method?

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#33

Post by echelon »

tgv wrote: post #29
abetusk wrote: post #9

As many know and point out, this idea is now very old (40+ years?). The big problem is that it creates "word salad" [0]. In the generative art and procgen community, the term "10,000 bowls of oatmeal" problem has been used [1]. Taking a step back, this is a perennial problem, even with AI old and new. I've heard that the early (good) chess bots, after Deep Blue, had a problem of only being locally context sensitive a…

Also: it works much better in English than in other languages I know. English grammar is simple, and many of the function words carry a strong prediction for the next tokens.

Which languages are harder for LLMs? Is there any writeup or analysis of this?

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#34

Post by tempodox »

abetusk wrote: post #9

As many know and point out, this idea is now very old (40+ years?). The big problem is that it creates "word salad" [0]. In the generative art and procgen community, the term "10,000 bowls of oatmeal" problem has been used [1]. Taking a step back, this is a perennial problem, even with AI old and new. I've heard that the early (good) chess bots, after Deep Blue, had a problem of only being locally context sensitive a…

And the logic salad problem will not be solved by LLMs because there is nothing in their construction that makes them capable of understanding logic.

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#35

Post by tempodox »

3abiton wrote: post #32
abetusk wrote: post #9

As many know and point out, this idea is now very old (40+ years?). The big problem is that it creates "word salad" [0]. In the generative art and procgen community, the term "10,000 bowls of oatmeal" problem has been used [1]. Taking a step back, this is a perennial problem, even with AI old and new. I've heard that the early (good) chess bots, after Deep Blue, had a problem of only being locally context sensitive a…

Any suggestiob on the background of the method?

Do a web search for “Markov chain”.

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#36

Post by [deleted] »

tempodox wrote: post #34
abetusk wrote: post #9

As many know and point out, this idea is now very old (40+ years?). The big problem is that it creates "word salad" [0]. In the generative art and procgen community, the term "10,000 bowls of oatmeal" problem has been used [1]. Taking a step back, this is a perennial problem, even with AI old and new. I've heard that the early (good) chess bots, after Deep Blue, had a problem of only being locally context sensitive a…

And the logic salad problem will not be solved by LLMs because there is nothing in their construction that makes them capable of understanding logic.

[deleted]

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#37

Post by moon_moon »

tempodox wrote: post #34
abetusk wrote: post #9

As many know and point out, this idea is now very old (40+ years?). The big problem is that it creates "word salad" [0]. In the generative art and procgen community, the term "10,000 bowls of oatmeal" problem has been used [1]. Taking a step back, this is a perennial problem, even with AI old and new. I've heard that the early (good) chess bots, after Deep Blue, had a problem of only being locally context sensitive a…

And the logic salad problem will not be solved by LLMs because there is nothing in their construction that makes them capable of understanding logic.

[dead]

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#38

Post by terminous »

mebassett wrote: post #22

this is old tech - but it had me thinking. Markov chains are picking the next token from a random set and they are giving approximately all possible tokens (from the training set) an equal probability. What if it weighted the probability - say using the inverse vector distance or cosine similarity of the neighbors as a proxy for probability, where the vector embedding came from word2vec...how close would the performa…

Yeah, that's just attention with extra steps

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#39

Post by tahnyall »

echelon wrote: post #33
tgv wrote: post #29

Earlier quoted context omitted.

Also: it works much better in English than in other languages I know. English grammar is simple, and many of the function words carry a strong prediction for the next tokens.

Which languages are harder for LLMs? Is there any writeup or analysis of this?

If I had to guess, from my time working with ASR (Advanced Speech Recognition) languages and its difficulty with the Indian (country of) accent; I'd guess Indian languages.

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#40

Post by YeGoblynQueenne »

abetusk wrote: post #9

As many know and point out, this idea is now very old (40+ years?). The big problem is that it creates "word salad" [0]. In the generative art and procgen community, the term "10,000 bowls of oatmeal" problem has been used [1]. Taking a step back, this is a perennial problem, even with AI old and new. I've heard that the early (good) chess bots, after Deep Blue, had a problem of only being locally context sensitive a…

The output is "word salad" because the probabilities of the model are uniform, albeit implicitly- eyballing the python, it selects the next n-gram uniformly at random. A better n-gram model would calculate the probability of an n-gram following another n-gram according to their frequency in the training corpus. With a larger corpus and better training techniques (some smoothing for one thing) you'd get much more coherent output.

And of course, if you trained a gigantic model on the entire web, then you'd get a very smooth approximation of the text in the corpus and very fluent, grammatical output. So basically, an LLM.

>> As many know and point out, this idea is now very old (40+ years?).

More like 110 years. Markov Chains were proposed in 1913 (by Markov) and popularised by Shannon in 1948 (in "A Mathematical Theory of Communication", the work that introduced information theory):

The underlying mathematics of the n-gram was first proposed by Markov (1913), who used what are now called Markov chains (bigrams and trigrams) to predict whether an upcoming letter in Pushkin’s Eugene Onegin would be a vowel or a con- sonant. Markov classified 20,000 letters as V or C and computed the bigram and trigram probability that a given letter would be a vowel given the previous one or two letters. Shannon (1948) applied n-grams to compute approximations to English word sequences. Based on Shannon’s work, Markov models were commonly used in engineering, linguistic, and psychological work on modeling word sequences by the 1950s. In a series of extremely influential papers starting with Chomsky (1956) and including Chomsky (1957) and Miller and Chomsky (1963), Noam Chomsky argued that “finite-state Markov processes”, while a possibly useful engineering heuristic, were incapable of being a complete cognitive model of human grammatical knowl- edge. These arguments led many linguists and computational linguists to ignore work in statistical modeling for decades.

https://web.archive.org/web/20220522005827/https://web.stanf...

(Note the usual shaking of angry fists at Chomsky. The NLP community continued on its path anyway, and built smooth, fluent generators of total bullshit and absolutely no progress towards a "cognitive model of human grammatical knowledge").

Post reply on HN