We uh as a group project did markov chains with some custom algorithmic adjustments, scraped reddit via api, and academic torrent (this data required a lot of cleaning and was orders of magnitude bigger, ran separately), to simulate posts and comments. In same group project we also implemented Variable Length Markov Chain, tree-based custom context-length, although a team member did the matrix based implementation custom context-length as well through some matrix magic.
Using a Markov chain to generate readable nonsense with 20 lines of Python
21–30 of 77 posts
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#22this is old tech - but it had me thinking. Markov chains are picking the next token from a random set and they are giving approximately all possible tokens (from the training set) an equal probability. What if it weighted the probability - say using the inverse vector distance or cosine similarity of the neighbors as a proxy for probability, where the vector embedding came from word2vec...how close would the performance be to a transformer model or even something like lstm rnns ? (I suppose I'm cheating a bit using word2vec here. I might as well just say I'm using the attention mechanism...)
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#23We had a bot that would randomly say things in our IRC channel 15 years ago that worked like this. You could also mention it to prompt it to reply. Every message was added to it's knowledge base and it would say random but hilarious stuff made up from all the nonsense we used to talk about. Good times.
For a hack day project someone took our IRC logs to do something similar for employees who had left (one bot per engineer, small company so a handful of bots). This made for a great afternoon until the bots started talking to each other and then started triggering the simple command the ops bot would accept. It quickly got shut down.
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#24We had a bot that would randomly say things in our IRC channel 15 years ago that worked like this. You could also mention it to prompt it to reply. Every message was added to it's knowledge base and it would say random but hilarious stuff made up from all the nonsense we used to talk about. Good times.
I wrote one bot like this. It was fun to play around.
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#25this is old tech - but it had me thinking. Markov chains are picking the next token from a random set and they are giving approximately all possible tokens (from the training set) an equal probability. What if it weighted the probability - say using the inverse vector distance or cosine similarity of the neighbors as a proxy for probability, where the vector embedding came from word2vec...how close would the performa…
That sounds interesting, but still fails to capture long-range dependencies.
If I understand correctly, what you're proposing is to replace co-occurrence frequency with word2vec cosine similarity.
I suppose it may help improve overall performance, you're still just blindly predicting the next word based on the previous one like a first order Markov chain would.
For example, it won't ever fit "2 plus 2 equals 4," because right when we get to equals, we discard all the previous words.
Perhaps if we could get the embedding model to consider the full sentence and then produce a set of probability-scored next token predictions it may work, but now we've just reinvented a transformer.
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#26> using Lewis Carroll’s Alice in Wonderland as input. The book is more or less nonsense to begin with,
Has he ever read Alice?
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#27This was posted 50 days ago - how does it reach first page again? https://news.ycombinator.com/from?site=benhoyt.com
By people being interested and upvoting it. Not everyone reads the home page all day every day.
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#28I was doing a similar experiment recently to generate random names that sound like names from a specific language. I was breaking the list of names apart into 3 and 2 letter parts, marking which fragments are from the start, middle and end. To generate the words I started from a random one from the start fragments, then continued with a random one from the middle fragments that starts with the latter that the previou…
I remember finding a set of english names generated with a markov chain, I wish I know how to make it, they sounded good.
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#29As many know and point out, this idea is now very old (40+ years?). The big problem is that it creates "word salad" [0]. In the generative art and procgen community, the term "10,000 bowls of oatmeal" problem has been used [1]. Taking a step back, this is a perennial problem, even with AI old and new. I've heard that the early (good) chess bots, after Deep Blue, had a problem of only being locally context sensitive a…
Also: it works much better in English than in other languages I know. English grammar is simple, and many of the function words carry a strong prediction for the next tokens.
Re: Using a Markov chain to generate readable nonsense with 20 lines of Python
#30this is old tech - but it had me thinking. Markov chains are picking the next token from a random set and they are giving approximately all possible tokens (from the training set) an equal probability. What if it weighted the probability - say using the inverse vector distance or cosine similarity of the neighbors as a proxy for probability, where the vector embedding came from word2vec...how close would the performa…
That sounds interesting, but still fails to capture long-range dependencies. If I understand correctly, what you're proposing is to replace co-occurrence frequency with word2vec cosine similarity. I suppose it may help improve overall performance, you're still just blindly predicting the next word based on the previous one like a first order Markov chain would. For example, it won't ever fit "2 plus 2 equals 4," beca…
> "I suppose it may help improve overall performance, you're still just blindly predicting the next word based on the previous one like a first order Markov chain would."
Instead of only taking in the last "token" as context to the function that generates the next token - take the last 15 tokens (ie. the last 2-3 sentences), and predict based on that. And that's your "attention" mechanism.