51–60 of 131 posts
Earlier quoted context omitted.
> I can share if you like Respectfully, absolutely nobody wants to read a copy-and-paste of a chat session with ChatGPT.
When you say nobody you mean you, right? You can't possible be answering for every single person in the world. I was having a discussion about similarities between Markov Chains and LLMs and short after I found this topic on HN, when I wrote "I can share if you like" was as a proof about the coincidence.
Earlier quoted context omitted.
The KV cache doesn't affect it because it's just an optimization. LLMs are stateless and don't take any other input than a fixed block of text. They don't have memory, which is the requirement for a Markov chain.
Have you ever actually worked with a basic markov problem? The markov property states that your state is a transition of probabilities entirely from the previous state. These states, inhabit a state space. The way you encode "memory" if you need it, e.g. say you need to remember if it rained the last 3 days, is by expanding said state space. In that case, you'd go from 1 state to 3 states, 2^3 states if you needed th…
Okay, so we're agreed.
I usually have this technical hypothetical discussions with ChatGpt, I can share if you like, me asking him this: aren't LLMs just huge Markov Chains?! And now I see your project... Funny
Don't know what happened. I stumbled onto a funny coincidence - me talking to a LLM about its similarities with MC - decided to share on a post about using MC to generate text. Got some nasty comments and a lot of down votes. Even though my comment sparked a pretty interesting discussion. Hate to be that guy, but I remember this place being nicer.
Everyone has access to ChatGPT. If we wanted its "opinion" we could ask it ourselves. Your offer is akin to "Hey everyone, want me to Google this and paste the results page here?". You would never offer to do that. Ask yourself why.
These posts are low-effort and add nothing to the conversation, yet the people who write them seem to expect everyone to be impressed by their contribution. If you can't understand why people find this irritating, I'm not sure what to tell you.
Cool article, it got me to play around with Markov models, too! I first did a Markov model over plain characters. > Itheve whe oiv v f vidleared ods alat akn atr. s m w bl po ar 20 Using pairs of consecutive characters (order-2 Markov model) helps, but not much: > I hateregratics.pyth fwd-i-sed wor is wors.py Triplets (order 3) are a bit better: > I Fed tooks of the say, I just train. All can beconsist answer efferes…
I did something similar many years ago. I fed about half a million words (two decades of mostly fantasy and science fiction writing) into a Markov model that could generate text using a “gram slider” ranging from 2-grams to 5-grams. I used it as a kind of “dream well” whenever I wanted to draw some muse from the same deep spring. It felt like a spiritual successor to what I used to do as a kid: flipping to a random p…
Did it work?
The problem with that is either your n-gram level is too low in which case it can't maintain any kind of cohesion, or your n-gram level is too high and it's basically just spitting out your existing corpus verbatim.
For me, I was more interested in something that could potentially combine two or three highly disparate concepts found in my previous works into a single outputted sentence - and then I would ideate upon it.
So I haven't opened the program in a long time so I just spun it up and generated a few outputs:
A giant baby is navel corked which if removed causes a vacuum.
Earlier quoted context omitted.
LLMs are indeed Markov chains. The breakthrough is that we are able to efficiently compute well performing probabilities for many states using ML.
LLMs are not Markov Chains unless you contort the meaning of a Markov Model State so much you could even include the human brain.