Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I fed 24 years of my blog posts to a Markov model

susam.net

Post reply on HN
Print view View original

21–30 of 131 posts

Re: I fed 24 years of my blog posts to a Markov model

#21

Post by davely »

vunderba wrote: post #5

I did something similar many years ago. I fed about half a million words (two decades of mostly fantasy and science fiction writing) into a Markov model that could generate text using a “gram slider” ranging from 2-grams to 5-grams. I used it as a kind of “dream well” whenever I wanted to draw some muse from the same deep spring. It felt like a spiritual successor to what I used to do as a kid: flipping to a random p…

I gave a talk in 2015 that did the same thing with my tweet history (about 20K at the time) and how I used it as source material for a Twitter bot that could reply to users. [1]

It was pretty fun!

[1] https://youtu.be/rMmXdiUGsr4

Re: I fed 24 years of my blog posts to a Markov model

#22

Post by chpatrick »

famouswaffles wrote: post #11
matusp wrote: post #9

Earlier quoted context omitted.

LLMs are indeed Markov chains. The breakthrough is that we are able to efficiently compute well performing probabilities for many states using ML.

LLMs are not Markov Chains unless you contort the meaning of a Markov Model State so much you could even include the human brain.

Not sure why that's contorting, a markov model is anything where you know the probability of going from state A to state B. The state can be anything. When it's text generation the state is previous text to text with an extra character, which is true for both LLMs and oldschool n-gram markov models.

Re: I fed 24 years of my blog posts to a Markov model

#23

Post by wizzwizz4 »

chpatrick wrote: post #22
famouswaffles wrote: post #11

Earlier quoted context omitted.

LLMs are not Markov Chains unless you contort the meaning of a Markov Model State so much you could even include the human brain.

Not sure why that's contorting, a markov model is anything where you know the probability of going from state A to state B. The state can be anything. When it's text generation the state is previous text to text with an extra character, which is true for both LLMs and oldschool n-gram markov models.

A GPT model would be modelled as an n-gram Markov model where n is the size of the context window. This is slightly useful for getting some crude bounds on the behaviour of GPT models in general, but is not a very efficient way to store a GPT model.

Re: I fed 24 years of my blog posts to a Markov model

#24

Post by famouswaffles »

chpatrick wrote: post #22
famouswaffles wrote: post #11

Earlier quoted context omitted.

LLMs are not Markov Chains unless you contort the meaning of a Markov Model State so much you could even include the human brain.

Not sure why that's contorting, a markov model is anything where you know the probability of going from state A to state B. The state can be anything. When it's text generation the state is previous text to text with an extra character, which is true for both LLMs and oldschool n-gram markov models.

Yes, technically you can frame an LLM as a Markov chain by defining the "state" as the entire sequence of previous tokens. But this is a vacuous observation under that definition, literally any deterministic or stochastic process becomes a Markov chain if you make the state space flexible enough. A chess game is a "Markov chain" if the state includes the full board position and move history. The weather is a "Markov chain" if the state includes all relevant atmospheric variables.

The problem is that this definition strips away what makes Markov models useful and interesting as a modeling framework. A “Markov text model” is a low-order Markov model (e.g., n-grams) with a fixed, tractable state and transitions based only on the last k tokens. LLMs aren’t that: they model using un-fixed long-range context (up to the window). For Markov chains, k is non-negotiable. It's a constant, not a variable. Once you make it a variable, near any process can be described as markovian, and the word is useless.

Re: I fed 24 years of my blog posts to a Markov model

#25

Post by fragmede »

swyx wrote: post #2

now i wonder if you can compare vs feeding into a GPT style transformer of a similar Order of Magnitude in param count..

That's the question today. Turns out transformers really are a leap forwards in terms of AI, whereas Markov chains, scaled up to today's level of resources and capacity, will still output gibberish.

Re: I fed 24 years of my blog posts to a Markov model

#26

Post by jacquesm »

hexnuts wrote: post #20

I just realized, one of the things that people might start doing is making a gamma model of their personality. I won't even approach who they were as a person, but it will give their Descendants (or bored researchers) a 60% approximation of who they were and their views. (60% is pulled from nowhere to justify my gamma designation, since there isn't a good scale for personality mirror quality for LLMs as far as I'm aw…

"Dixie can't meaningfully grow as a person. All that he ever will be is burned onto that cart;"

"Do me a favor, boy. This scam of yours, when it's over, you erase this god-damned thing."

Re: I fed 24 years of my blog posts to a Markov model

#27

Post by idiotsecant »

vunderba wrote: post #5

I did something similar many years ago. I fed about half a million words (two decades of mostly fantasy and science fiction writing) into a Markov model that could generate text using a “gram slider” ranging from 2-grams to 5-grams. I used it as a kind of “dream well” whenever I wanted to draw some muse from the same deep spring. It felt like a spiritual successor to what I used to do as a kid: flipping to a random p…

Did it work?

Re: I fed 24 years of my blog posts to a Markov model

#28

Post by chpatrick »

famouswaffles wrote: post #24
chpatrick wrote: post #22

Earlier quoted context omitted.

Not sure why that's contorting, a markov model is anything where you know the probability of going from state A to state B. The state can be anything. When it's text generation the state is previous text to text with an extra character, which is true for both LLMs and oldschool n-gram markov models.

Yes, technically you can frame an LLM as a Markov chain by defining the "state" as the entire sequence of previous tokens. But this is a vacuous observation under that definition, literally any deterministic or stochastic process becomes a Markov chain if you make the state space flexible enough. A chess game is a "Markov chain" if the state includes the full board position and move history. The weather is a "Markov…

Sure many things can be modelled as Markov chains, which is why they're useful. But it's a mathematical model so there's no bound on how big the state is allowed to be. The only requirement is that all you need is the current state to determine the probabilities of the next state, which is exactly how LLMs work. They don't remember anything beyond the last thing they generated. They just have big context windows.

Re: I fed 24 years of my blog posts to a Markov model

#29

Post by chpatrick »

wizzwizz4 wrote: post #23
chpatrick wrote: post #22

Earlier quoted context omitted.

Not sure why that's contorting, a markov model is anything where you know the probability of going from state A to state B. The state can be anything. When it's text generation the state is previous text to text with an extra character, which is true for both LLMs and oldschool n-gram markov models.

A GPT model would be modelled as an n-gram Markov model where n is the size of the context window. This is slightly useful for getting some crude bounds on the behaviour of GPT models in general, but is not a very efficient way to store a GPT model.

I'm not saying it's an n-gram Markov model or that you should store them as a lookup table. Markov models are just a mathematical concept that don't say anything about storage, just that the state change probabilities are a pure function of the current state.

Re: I fed 24 years of my blog posts to a Markov model

#30

Post by famouswaffles »

chpatrick wrote: post #28
famouswaffles wrote: post #24

Earlier quoted context omitted.

Yes, technically you can frame an LLM as a Markov chain by defining the "state" as the entire sequence of previous tokens. But this is a vacuous observation under that definition, literally any deterministic or stochastic process becomes a Markov chain if you make the state space flexible enough. A chess game is a "Markov chain" if the state includes the full board position and move history. The weather is a "Markov…

Sure many things can be modelled as Markov chains, which is why they're useful. But it's a mathematical model so there's no bound on how big the state is allowed to be. The only requirement is that all you need is the current state to determine the probabilities of the next state, which is exactly how LLMs work. They don't remember anything beyond the last thing they generated. They just have big context windows.

>Sure many things can be modelled as Markov chains

Again, no they can't, unless you break the definition. K is not a variable. It's as simple as that. The state cannot be flexible.

1. The markov text model uses k tokens, not k tokens sometimes, n tokens other times and whatever you want it to be the rest of the time.

2. A markov model is explcitly described as 'assuming that future states depend only on the current state, not on the events that occurred before it'. Defining your 'state' such that every event imaginable can be captured inside it is a 'clever' workaround, but is ultimately describing something that is decidedly not a markov model.

Post reply on HN