Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using a Markov chain to generate readable nonsense with 20 lines of Python

benhoyt.com

Post reply on HN
Print view View original

51–60 of 77 posts

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#51

Post by dahart »

See also Mark V Shaney https://en.wikipedia.org/wiki/Mark_V._Shaney

This was a class assignment in college, we had a lot of fun with it, and one of my classmates, the brilliant Alyosha Efros, decided to apply the exact same technique to images instead of text. It turned into a paper that revitalized texture synthesis in the Siggraph community. The most interesting part about it (in my opinion) is that he ran it on images of text, and it produces images of readable text nonsense! With the right window size, perhaps there’s a nonzero possibility of being able to produce the same text either way. This always struck me as very meta and makes me wonder if there are ways we could go in reverse with image processing (or other types of data); if there’s a latent underlying representation for the information contained in image content that is much smaller and more efficient to work with. Neural networks might or might not be providing evidence.

https://people.eecs.berkeley.edu/~efros/research/EfrosLeung....

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#52

Post by Frummy »

Frummy wrote: post #21

We uh as a group project did markov chains with some custom algorithmic adjustments, scraped reddit via api, and academic torrent (this data required a lot of cleaning and was orders of magnitude bigger, ran separately), to simulate posts and comments. In same group project we also implemented Variable Length Markov Chain, tree-based custom context-length, although a team member did the matrix based implementation cu…

Genuinely curious to hear what’s wrong with what I thought a cool, curious fun time implementing markov chains. We learned a lot about statistics and linear algebra, matrix diagonalisation and exponentiation, I find it very relevant and can answer questions if you don’t understand.

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#53

Post by CamperBob2 »

stabbles wrote: post #13
max_ wrote: post #10

I wonder how this would behave if trained on 1 trillion words like the LLMs. Also, Is training a Markov cheaper that training neural nets? It would be a great way to cut AI costs if they could be made as effective as neural nets. There is also an interesting post by Orange duck. [0] [0] https://theorangeduck.com/page/17-line-markov-chain

Yeah, a century old approach using frequency tables where the next word is generated based just on the previous two words is highly competitive with contemporary neural nets. To produce coherent sentences, do you ever need to remember more than the last two words you said? I doubt it.

Now ask it to render a deepfake of Noam Chomsky waving his hands saying "Nothing to see here."

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#54

Post by doubloon »

abetusk wrote: post #9

As many know and point out, this idea is now very old (40+ years?). The big problem is that it creates "word salad" [0]. In the generative art and procgen community, the term "10,000 bowls of oatmeal" problem has been used [1]. Taking a step back, this is a perennial problem, even with AI old and new. I've heard that the early (good) chess bots, after Deep Blue, had a problem of only being locally context sensitive a…

i put Alexis De Tocqueville through this and the results were legible but hard to read.

Then I put a demagogue politician speech through this and the results were almost like a speech that politician would have given. It was hard to tell the difference between the original and the generated.

in other words, if the public admires word salad, and someone speaks in word salad, then word salad output would not be considered as problematic by a large number of people.

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#55

Post by sgt101 »

gattilorenz wrote: post #19
sgt101 wrote: post #17

Earlier quoted context omitted.

well - the lawyers from the NYT called, and they disagree...

I didn’t get the call, so it’s not clear what they disagree on :) I doubt the have anything to say about Markov chains. We’re talking about technical possibilities, not legality of training corpora

The NN's trained on huge corpus's appear to be prone to memorization as well.

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#56

Post by AndruLuvisi »

The first edition of Programming Perl had a script that did this called travesty. You can download it from https://resources.oreilly.com/examples/9780937175644

When I was learning Perl in the '90s, I remember having a lot of fun running this on different manual pages.

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#57

Post by chuckadams »

doubloon wrote: post #54
abetusk wrote: post #9

As many know and point out, this idea is now very old (40+ years?). The big problem is that it creates "word salad" [0]. In the generative art and procgen community, the term "10,000 bowls of oatmeal" problem has been used [1]. Taking a step back, this is a perennial problem, even with AI old and new. I've heard that the early (good) chess bots, after Deep Blue, had a problem of only being locally context sensitive a…

i put Alexis De Tocqueville through this and the results were legible but hard to read. Then I put a demagogue politician speech through this and the results were almost like a speech that politician would have given. It was hard to tell the difference between the original and the generated. in other words, if the public admires word salad, and someone speaks in word salad, then word salad output would not be conside…

I recall one site way back that used a Markov chain generator to mash up Karl Marx and Ayn Rand. Was fairly plausible reading, actually.

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#58

Post by chuckadams »

AndruLuvisi wrote: post #56

The first edition of Programming Perl had a script that did this called travesty. You can download it from https://resources.oreilly.com/examples/9780937175644 When I was learning Perl in the '90s, I remember having a lot of fun running this on different manual pages.

It’s also built into emacs: I believe it’s called disassociated-press

Re: Using a Markov chain to generate readable nonsense with 20 lines of Python

#59

Post by heikkilevanto »

I remember as a school boy I heard about Markov Chains, and played a bit with them. Instead of building big n-gram tables, I found a simpler way. Start with one n-gram, then scan the source text until you find the next occurrence of it, and take the next letter after it. Output it, and append it to your n-gram, dropping its first letter, and repeat until you have sufficient amount of nonsense text.
Post reply on HN