Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I fed 24 years of my blog posts to a Markov model

susam.net

Post reply on HN
Print view View original

91–100 of 131 posts

Re: I fed 24 years of my blog posts to a Markov model

#91

Post by srean »

matusp wrote: post #90
srean wrote: post #85

Earlier quoted context omitted.

They are definitely not Markov Chains they may, however, be Markov Models . There's a difference between MC and MM.

What do you mean? The states are fully observable (current array of tokens), and using an LLM we calculate the probabilities of moving between them. What is not MC about this?

I suggest getting familiar with or brushing up on the differences between a Markov Chain and a Markov Model. The former is a substantial restriction of the latter. The classic by Kemeny and Snell is a good readable reference.

MC have constant and finite context length, their state is the most recent k tuple of emitted alphabets and transition probabilities are invariant (to time and tokens emitted)

Re: I fed 24 years of my blog posts to a Markov model

#92

Post by travisjungroth »

yard2010 wrote: post #75
sebastianmestre wrote: post #34

Cool article, it got me to play around with Markov models, too! I first did a Markov model over plain characters. > Itheve whe oiv v f vidleared ods alat akn atr. s m w bl po ar 20 Using pairs of consecutive characters (order-2 Markov model) helps, but not much: > I hateregratics.pyth fwd-i-sed wor is wors.py Triplets (order 3) are a bit better: > I Fed tooks of the say, I just train. All can beconsist answer efferes…

Reading this I get this weird feeling that something there is trying to communicate, which is equally horrifying as the alternative - we are alone, our minds are trying to find order in chaos, there is no meaning except what we create.

The alternative to something trying to communicate through a Markov model isn’t that we’re alone. Just because there’s no life on Mars doesn’t mean there’s no other life in the universe.

Re: I fed 24 years of my blog posts to a Markov model

#93

Post by chpatrick »

famouswaffles wrote: post #76
saithound wrote: post #70

Earlier quoted context omitted.

Just a heads-up: this is not the first time somebody has to explain Markov chains to famouswaffles on HN, and I'm pretty sure it won't be the last. Engaging further might not be worth it.

I did not even remember you and had to dig to find out what you were on about. Just a heads up, if you've had a previous argument and you want to bring that up later then just speak plainly. Why act like "somebody" is anyone but you? My response to both of you is the same. LLMs do depend on previous events, but you say they don't because you've redefined state to include previous events. It's a circular argument. In…

By that definition n-gram Markov chain text generators also include previous state because you always put the last n grams. :) It's exactly the same situation as LLMs, just with higher, but still fixed n.

Re: I fed 24 years of my blog posts to a Markov model

#94

Post by sebastianmestre »

yard2010 wrote: post #75
sebastianmestre wrote: post #34

Cool article, it got me to play around with Markov models, too! I first did a Markov model over plain characters. > Itheve whe oiv v f vidleared ods alat akn atr. s m w bl po ar 20 Using pairs of consecutive characters (order-2 Markov model) helps, but not much: > I hateregratics.pyth fwd-i-sed wor is wors.py Triplets (order 3) are a bit better: > I Fed tooks of the say, I just train. All can beconsist answer efferes…

Reading this I get this weird feeling that something there is trying to communicate, which is equally horrifying as the alternative - we are alone, our minds are trying to find order in chaos, there is no meaning except what we create.

I had the same feeling while testing the code. It might be caused by seeing the increasingly coherent output of the different models, makes you feel like it's getting smarter.

Re: I fed 24 years of my blog posts to a Markov model

#95

Post by [deleted] »

chpatrick wrote: post #93
famouswaffles wrote: post #76

Earlier quoted context omitted.

I did not even remember you and had to dig to find out what you were on about. Just a heads up, if you've had a previous argument and you want to bring that up later then just speak plainly. Why act like "somebody" is anyone but you? My response to both of you is the same. LLMs do depend on previous events, but you say they don't because you've redefined state to include previous events. It's a circular argument. In…

By that definition n-gram Markov chain text generators also include previous state because you always put the last n grams. :) It's exactly the same situation as LLMs, just with higher, but still fixed n.

[deleted]

Re: I fed 24 years of my blog posts to a Markov model

#96

Post by mattacular »

user_7832 wrote: post #72

I can’t believe no one’s mentioned the Harry Potter fanfic written by a Markov Chain. If you’re familiar with HP, I highly recommend reading Harry Potter and the Portrait of What Looked Like a Large Pile of Ash. Here’s a link: https://botnik.org/content/harry-potter.html

Genuine question: Why would anyone want to read that? I glanced at the first sentence and decided not to go any further.

It is hollow text. It has no properties of what I'd want to get out of even the worst book produced by human minds.

Even more sophisticated models have a ceiling of pablum.

Re: I fed 24 years of my blog posts to a Markov model

#97

Post by grahamnorton39 »

mattacular wrote: post #96
user_7832 wrote: post #72

I can’t believe no one’s mentioned the Harry Potter fanfic written by a Markov Chain. If you’re familiar with HP, I highly recommend reading Harry Potter and the Portrait of What Looked Like a Large Pile of Ash. Here’s a link: https://botnik.org/content/harry-potter.html

Genuine question: Why would anyone want to read that? I glanced at the first sentence and decided not to go any further. It is hollow text. It has no properties of what I'd want to get out of even the worst book produced by human minds. Even more sophisticated models have a ceiling of pablum.

While hollow, it is also bad (and absurd) enough to be quite entertaining. It’s from an era where this wasn’t far off the state of the art for coming up with machine-generated text—context that makes it quite a bit funnier than if it were generated by an LLM today.

That said, it’s obviously not to everyone’s tastes!

Re: I fed 24 years of my blog posts to a Markov model

#98

Post by lacunary »

frumiousirc wrote: post #89
lacunary wrote: post #4

I recall a Markov chain bot on IRC in the mid 2000s. I didn't see anything better until gpt came along!

Perhaps you are thinking of megahal https://homepage.kranzky.com/megahal/Index.html or if a bit later in the millennium, cobe https://teichman.org/blog/

ah, probably so, looks like there were eggdrop scripts for megahal, thanks!

Re: I fed 24 years of my blog posts to a Markov model

#99

Post by kqr »

vunderba wrote: post #56
idiotsecant wrote: post #27

Earlier quoted context omitted.

Did it work?

So that's the key difference. A lot of people train these Markov models with the expectation that they're going to be able to use the generated output in isolation. The problem with that is either your n-gram level is too low in which case it can't maintain any kind of cohesion, or your n-gram level is too high and it's basically just spitting out your existing corpus verbatim. For me, I was more interested in someth…

> A giant baby is navel corked which if removed causes a vacuum.

Very The Age of Wire and String.

Re: I fed 24 years of my blog posts to a Markov model

#100

Post by famouswaffles »

chpatrick wrote: post #93
famouswaffles wrote: post #76

Earlier quoted context omitted.

I did not even remember you and had to dig to find out what you were on about. Just a heads up, if you've had a previous argument and you want to bring that up later then just speak plainly. Why act like "somebody" is anyone but you? My response to both of you is the same. LLMs do depend on previous events, but you say they don't because you've redefined state to include previous events. It's a circular argument. In…

By that definition n-gram Markov chain text generators also include previous state because you always put the last n grams. :) It's exactly the same situation as LLMs, just with higher, but still fixed n.

We've been through this. The context of a LLM is not fixed. Context windows =/ n gram orders.

They don't because n gram orders are too small and rigid to include the history in the general case.

I think srean's comment up the thread is spot on. This current situation where the state can be anything you want it to be just does not make a productive conversation.

Post reply on HN