I fed 24 years of my blog posts to a Markov model
121–130 of 131 posts
Re: I fed 24 years of my blog posts to a Markov model
#122In the discussion thread there seems to be a somewhat contentious argument about what is a Markov Model, whether LLMs are one, RNNs are one and so on. Markov Models are anything that has state and emit tokens based only on its current state and undergoes a state transition. The token emission and state transitions are usually probabilistic -- a statistical/probabilistic analogue of a state machine. The deterministic…
This is right in terms of the rigorous statistical sense of “Markov model”. But in practice in the world of NLP and chatbots through the 90s and 2000s, “Markov model” was usually used to refer to Markov chains (ie you only condition on the previous k words). Hence the term “Hidden Markov Model” to refer to what you’re calling a Markov model.
Re: I fed 24 years of my blog posts to a Markov model
#123I think this is more correctly described as a trigram model than a Markov model, if it would naturally expand to 4-grams when they were available, etc, the text would look more coherent Iirc there was some research on "infini-gram", that is a very large ngram model, that allegedly got performance close to LLMs in some domains a couple years back
Google made some very large ngram models around twenty years ago. This being before the era of ultra-high-speed internet, it was distributed as a set of 6 DVDs. It achieved state-of-the-art performance at tasks like spelling correction at the time. However, unlike an LLM, it can't generalize at all; if an n-gram isn't in the training corpus it has no idea how to handle it. https://research.google/blog/all-our-n-gram-…
I have this DVD set in my basement. Technically, there are still methods for estimating the probability of unseen ngrams. Backoff (interpolating with lower grams) is an option. You can also impose prior distributions like a Bayesian so that you can make "rational" guesses.
Ngrams are surprisingly powerful for how little computation they require. They can be trained in seconds even with tons of data.
Re: I fed 24 years of my blog posts to a Markov model
#124I can’t believe no one’s mentioned the Harry Potter fanfic written by a Markov Chain. If you’re familiar with HP, I highly recommend reading Harry Potter and the Portrait of What Looked Like a Large Pile of Ash. Here’s a link: https://botnik.org/content/harry-potter.html
That reads as coherently as a pile of Zen koans.
Re: I fed 24 years of my blog posts to a Markov model
#125In the discussion thread there seems to be a somewhat contentious argument about what is a Markov Model, whether LLMs are one, RNNs are one and so on. Markov Models are anything that has state and emit tokens based only on its current state and undergoes a state transition. The token emission and state transitions are usually probabilistic -- a statistical/probabilistic analogue of a state machine. The deterministic…
This is right in terms of the rigorous statistical sense of “Markov model”. But in practice in the world of NLP and chatbots through the 90s and 2000s, “Markov model” was usually used to refer to Markov chains (ie you only condition on the previous k words). Hence the term “ Hidden Markov Model” to refer to what you’re calling a Markov model.
I covered hidden Markov Models briefly in my comment.
It depends on whether the state is visible in the observations or not. Hidden or not is an orthogonal axis of variation compared to the other variations mentioned in the comment.
In a non-hidden model there is no ambiguity or uncertainty about what the current state is.
Re: I fed 24 years of my blog posts to a Markov model
#126So, are current LLMs better because artificial neural networks are better predictors than Markov models, or because of the scale of the training data? Just putting it out there..
Markov models usually only predict the next token given the two preceding tokens (trigram model) because the data gets so exceptionally sparse beyond that, that it becomes impossible to make probability estimations (despite back-off, smoothing, etc.). I recommend you to read Bengio et al.’s 2003 paper which describes this issue in more detail and introduces distributional representations (embeddings) in an RNN to avo…
> Markov models usually only predict the next token given the two preceding tokens (trigram model) because the data gets so exceptionally sparse beyond that
Of course, that's because it is a probability along a single dimension with a chain-length along that one dimension while LLMs and NNs use multiple dimensions (They are meshed, not chained).
I really want to know what the result would look like with a few more dimensions resulting in a markov mesh type structure rather than a chain structure.
Re: I fed 24 years of my blog posts to a Markov model
#127Earlier quoted context omitted.
I think you're confusing Markov chains and "Markov chain text generators". A Markov chain is a mathematical structure where the probabilities of going to the next state only depend on the current state and not the previous path taken. That's it. It doesn't say anything about whether the probabilities are computed by a transformer or stored in a lookup table, it just exists. How the probabilities are determined in a p…
'A Markov chain is a mathematical structure where the probabilities of going to the next state only depend on the current state and not the previous path taken.' My point, which seems so hard to grasp for whatever reason is that In a Markov chain, state is a well defined thing. It's not a variable you can assign any property to. LLMs do depend on the previous path taken. That's the entire reason they're so useful! An…
They only have the previous path in as much as n-gram Markov text generators have the previous path.
Re: I fed 24 years of my blog posts to a Markov model
#128It laughed and gave him a kiss
Re: I fed 24 years of my blog posts to a Markov model
#129Earlier quoted context omitted.
It's not n sometimes, k tokens some other times. LLMs have fixed context windows, you just sometimes have less text so it's not full. They're pure functions from a fixed size block of text to a probability distribution of the next character, same as the classic lookup table n gram Markov chain model.
1. A context limit is not a Markov order. An n-gram model’s defining constraint is: there exists a small constant k such that the next-token distribution depends only on the last k tokens, full stop. You can't use a k-trained markov model on anything but k tokens, and each token has the same relationship with each other regardless. An LLM’s defining behavior is the opposite: within its window it can condition on any…
> An n-gram model’s defining constraint is: there exists a small constant k such that the next-token distribution depends only on the last k tokens, full stop.
I don't necessarily agree with GP, but I also don't think that a markov chain and markov generator definitions include the word "small".
That constant can be as large as you need it to be.
Re: I fed 24 years of my blog posts to a Markov model
#130Earlier quoted context omitted.
Thanks for the reference and I stand corrected. And yes I had looked at it a long time ago and will give it another read. But I think it is saying that RNNs are a means of approximating a statistical property of a collection of text. That property is what we today think of as "completion"? That is, glorified auto complete, and not "distributed representations" of the world. Would you agree?
distributed representations Distributional representations, not distributed. https://en.wikipedia.org/wiki/Distributional_semantics#Distr...
This is the problem. I am arguing there are no distributed representations (cf Harnad's original symbol grounding problem paper, Hinton, and others). There are "distributional representations" by definition (cf the Wikipedia entry)