Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I fed 24 years of my blog posts to a Markov model

susam.net

Post reply on HN
Print view View original

101–110 of 131 posts

Re: I fed 24 years of my blog posts to a Markov model

#101

Post by user_7832 »

mattacular wrote: post #96
user_7832 wrote: post #72

I can’t believe no one’s mentioned the Harry Potter fanfic written by a Markov Chain. If you’re familiar with HP, I highly recommend reading Harry Potter and the Portrait of What Looked Like a Large Pile of Ash. Here’s a link: https://botnik.org/content/harry-potter.html

Genuine question: Why would anyone want to read that? I glanced at the first sentence and decided not to go any further. It is hollow text. It has no properties of what I'd want to get out of even the worst book produced by human minds. Even more sophisticated models have a ceiling of pablum.

It's purely meant to be an absurdist read. It obviously makes no sense, yet is close enough to actual language patterns for (some, at least) people to find it hilarious. I had tears in my eyes from laughing way too hard when I first read it.

Re: I fed 24 years of my blog posts to a Markov model

#102

Post by NuclearPM »

srean wrote: post #79

In the discussion thread there seems to be a somewhat contentious argument about what is a Markov Model, whether LLMs are one, RNNs are one and so on. Markov Models are anything that has state and emit tokens based only on its current state and undergoes a state transition. The token emission and state transitions are usually probabilistic -- a statistical/probabilistic analogue of a state machine. The deterministic…

Learnable? What does that mean?

Re: I fed 24 years of my blog posts to a Markov model

#103

Post by srean »

NuclearPM wrote: post #102
srean wrote: post #79

In the discussion thread there seems to be a somewhat contentious argument about what is a Markov Model, whether LLMs are one, RNNs are one and so on. Markov Models are anything that has state and emit tokens based only on its current state and undergoes a state transition. The token emission and state transitions are usually probabilistic -- a statistical/probabilistic analogue of a state machine. The deterministic…

Learnable? What does that mean?

There are various notions of this.

The most basic/naive one is where one can estimate the unknown parameters of the model given example token streams generated by the model.

Re: I fed 24 years of my blog posts to a Markov model

#104

Post by matusp »

srean wrote: post #91
matusp wrote: post #90

Earlier quoted context omitted.

What do you mean? The states are fully observable (current array of tokens), and using an LLM we calculate the probabilities of moving between them. What is not MC about this?

I suggest getting familiar with or brushing up on the differences between a Markov Chain and a Markov Model. The former is a substantial restriction of the latter. The classic by Kemeny and Snell is a good readable reference. MC have constant and finite context length , their state is the most recent k tuple of emitted alphabets and transition probabilities are invariant (to time and tokens emitted)

LLMs definitely also have finite context length. And if we consider padding, it is also constant. The k is huge compared to most Markov chains used historically, but it doesn't make it less finite.

Re: I fed 24 years of my blog posts to a Markov model

#106

Post by pavel_lishin »

atum47 wrote: post #45
atum47 wrote: post #3

I usually have this technical hypothetical discussions with ChatGpt, I can share if you like, me asking him this: aren't LLMs just huge Markov Chains?! And now I see your project... Funny

Don't know what happened. I stumbled onto a funny coincidence - me talking to a LLM about its similarities with MC - decided to share on a post about using MC to generate text. Got some nasty comments and a lot of down votes. Even though my comment sparked a pretty interesting discussion. Hate to be that guy, but I remember this place being nicer.

Nobody was being nasty. roarcher explained why people reacted the way they did.

Re: I fed 24 years of my blog posts to a Markov model

#107

Post by srean »

matusp wrote: post #104
srean wrote: post #91

Earlier quoted context omitted.

I suggest getting familiar with or brushing up on the differences between a Markov Chain and a Markov Model. The former is a substantial restriction of the latter. The classic by Kemeny and Snell is a good readable reference. MC have constant and finite context length , their state is the most recent k tuple of emitted alphabets and transition probabilities are invariant (to time and tokens emitted)

LLMs definitely also have finite context length. And if we consider padding, it is also constant. The k is huge compared to most Markov chains used historically, but it doesn't make it less finite.

That's not correct. Even a toy like an exponential weighted moving averaging produces unbounded context (of diminishing influence).

Re: I fed 24 years of my blog posts to a Markov model

#108

Post by matusp »

srean wrote: post #107
matusp wrote: post #104

Earlier quoted context omitted.

LLMs definitely also have finite context length. And if we consider padding, it is also constant. The k is huge compared to most Markov chains used historically, but it doesn't make it less finite.

That's not correct. Even a toy like an exponential weighted moving averaging produces unbounded context (of diminishing influence).

What do you mean? I can only input k tokens into my LLM to calculate the probs. That is the definition of my state. In the exact way that N-gram LMs use N tokens, but instead of using ML models, they calculate the probabilities based on observed frequencies. There is no unbounded context anywhere.

Re: I fed 24 years of my blog posts to a Markov model

#109

Post by srean »

matusp wrote: post #108
srean wrote: post #107

Earlier quoted context omitted.

That's not correct. Even a toy like an exponential weighted moving averaging produces unbounded context (of diminishing influence).

What do you mean? I can only input k tokens into my LLM to calculate the probs. That is the definition of my state. In the exact way that N-gram LMs use N tokens, but instead of using ML models, they calculate the probabilities based on observed frequencies. There is no unbounded context anywhere.

That's different.

You can certainly feed k-grams one at a time to, estimate the the probability distribution over next token and use that to simulate a Markov Chain and reinitialize the LLM (drop context). In this process the LLM is just a look up table to simulate your MC.

But an LLM on its own doesn't drop context to generate, it's transition probabilities change depending on the tokens.

Re: I fed 24 years of my blog posts to a Markov model

#110

Post by pessimizer »

You could literally buy this at Egghead software for $3 from the bargain bin in 1992. I know, because I did. I fed it 5 years worth of my juvenile rants, and laughed at how pompous I sounded through a blender.

https://archive.org/details/Babble_1020

A fairly prescient example of how long ago 4 years was:

https://forum.winworldpc.com/discussion/12953/software-spotl...

Post reply on HN