41–50 of 77 posts
Earlier quoted context omitted.
Also: it works much better in English than in other languages I know. English grammar is simple, and many of the function words carry a strong prediction for the next tokens.
Which languages are harder for LLMs? Is there any writeup or analysis of this?
As many know and point out, this idea is now very old (40+ years?). The big problem is that it creates "word salad" [0]. In the generative art and procgen community, the term "10,000 bowls of oatmeal" problem has been used [1]. Taking a step back, this is a perennial problem, even with AI old and new. I've heard that the early (good) chess bots, after Deep Blue, had a problem of only being locally context sensitive a…
The output is "word salad" because the probabilities of the model are uniform, albeit implicitly- eyballing the python, it selects the next n-gram uniformly at random. A better n-gram model would calculate the probability of an n-gram following another n-gram according to their frequency in the training corpus. With a larger corpus and better training techniques (some smoothing for one thing) you'd get much more cohe…
I don't understand what you mean here, isn't that exactly what an n-gram model is already designed to achieve where n > 1? Since this is a 2-gram model isn't it already doing this?
Earlier quoted context omitted.
Also: it works much better in English than in other languages I know. English grammar is simple, and many of the function words carry a strong prediction for the next tokens.
Which languages are harder for LLMs? Is there any writeup or analysis of this?
As many know and point out, this idea is now very old (40+ years?). The big problem is that it creates "word salad" [0]. In the generative art and procgen community, the term "10,000 bowls of oatmeal" problem has been used [1]. Taking a step back, this is a perennial problem, even with AI old and new. I've heard that the early (good) chess bots, after Deep Blue, had a problem of only being locally context sensitive a…
The output is "word salad" because the probabilities of the model are uniform, albeit implicitly- eyballing the python, it selects the next n-gram uniformly at random. A better n-gram model would calculate the probability of an n-gram following another n-gram according to their frequency in the training corpus. With a larger corpus and better training techniques (some smoothing for one thing) you'd get much more cohe…
From a quick look, it doesn't seem to sample uniformly. w3 is added to a list for the context w1, w2, not a set. So, say word A occurs twice as often in a particular context as B, it will be in the list twice as often. So, even though a uniform choice function is used, the probability of A getting samples is twice as high.
You get a word salad because a trigram model has to little context to do anything else. This is a well-known issue with Markov and hidden Markov models.
(Fun fact: some hidden Markov model taggers switch to a different trigram distribution for V2 languages after seeing the finite verb, otherwise they often fail catastrophically in the verb cluster due to the limited context.)
I was doing a similar experiment recently to generate random names that sound like names from a specific language. I was breaking the list of names apart into 3 and 2 letter parts, marking which fragments are from the start, middle and end. To generate the words I started from a random one from the start fragments, then continued with a random one from the middle fragments that starts with the latter that the previou…
I remember finding a set of english names generated with a markov chain, I wish I know how to make it, they sounded good.
You can learn! I know it's easy to say something is easy to a beginner, but figuring out Markov chains is truly something you can get the basics of over a weekend if you've ever written any software at all.
As many know and point out, this idea is now very old (40+ years?). The big problem is that it creates "word salad" [0]. In the generative art and procgen community, the term "10,000 bowls of oatmeal" problem has been used [1]. Taking a step back, this is a perennial problem, even with AI old and new. I've heard that the early (good) chess bots, after Deep Blue, had a problem of only being locally context sensitive a…
And the logic salad problem will not be solved by LLMs because there is nothing in their construction that makes them capable of understanding logic.
As a total handwave, I would expect an LLM trained on formal logic and a huge corpus of proofs to produce pretty strong logic output.
What if you trained a markov chain and got weights of every pair or sequence of pairs for a certain length.
Could you also do rotations with this information in vector space? With multiple points of freedom?
What if you could do inverse kinematics with markov chains?
Like robot planning for a goal?
I wonder how this would behave if trained on 1 trillion words like the LLMs. Also, Is training a Markov cheaper that training neural nets? It would be a great way to cut AI costs if they could be made as effective as neural nets. There is also an interesting post by Orange duck. [0] [0] https://theorangeduck.com/page/17-line-markov-chain
The only way to make it sound less like a word salad is to increase the n-gram length (e.g. use the last 10 words to predict the next), but at that point it starts repeating the corpus. I guess technically you can train on a huge corpus like those of NNs to mitigate that, but you’ll end up with more refined word salads then (edit: refined as in “locally coherent but globally still a word salad”)
https://dl.acm.org/doi/10.5555/944919.944966
I recommend everyone interested in neural language models and/or wondering why we can't scale up Markov models by just using longer ngrams to read this paper. It's pretty accessible and explains how we got to where we are now in 2023.
This is exactly the approach I used back in the late 80s; I published a Markov chain text generator on comp.sources.games that digested Usenet posts, built tables and used the previous two words to generate the next word. It was mildly amusing at the time. Someone resurrected it and put on GitHub, which I'm fine with.
Mark V. Shaney?