Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are you better than a language model at predicting the next word?

joel.tools

Post reply on HN
Print view View original

31–40 of 105 posts

Re: Are you better than a language model at predicting the next word?

#31

Post by sigbottle »

JoelEinbinder wrote: post #19
stackghost wrote: post #18

This is just a test of how likely you are to generate the same word as the LLM . The LLM does not produce the "correct" next word as there are multiple correct words that fit grammatically and can be used to continue the sentence while maintaining context. I don't see what this has to do with being "smarter" than anything. Example: 1. I see a business decision here. Arm cores have licensing fees attached to them. Arm…

The LLM didn’t generate the next word. Hacker News commenters did. You can see the source of the comment on the results screen.

Do LLM's generate words on the fly or can they sort of "go back" and correct themselves? stackghost brought up a good point I didn't think about before

Re: Are you better than a language model at predicting the next word?

#32

Post by lostmsu »

JoelEinbinder wrote: post #26
lostmsu wrote: post #21

I think this is a good joke on nay-sayers. But if author is here, I would like a clarification if user is picking the next token or the next word? Cause if it is the latter, I think this test is invalid.

The language model generating the candidate answers generates tokens until a full word is produced. The language models picking their answer choose the completion that results in the lowest perplexity independent of the tokenization.

I'd say the test is still not quite valid, and more of in between the original "valid" task and "guess what LLM would say" as suggested in another comment here. The reason is: it might be easier for LLMs to choose the completion out of their own generated variants (1) than the real token distribution.

1. perhaps even out of variants generated by other LLMs

Re: Are you better than a language model at predicting the next word?

#33

Post by DiscourseFan »

JoelEinbinder wrote: post #19
stackghost wrote: post #18

This is just a test of how likely you are to generate the same word as the LLM . The LLM does not produce the "correct" next word as there are multiple correct words that fit grammatically and can be used to continue the sentence while maintaining context. I don't see what this has to do with being "smarter" than anything. Example: 1. I see a business decision here. Arm cores have licensing fees attached to them. Arm…

The LLM didn’t generate the next word. Hacker News commenters did. You can see the source of the comment on the results screen.

At this point, we've all gotten quite used to the "style" of LLM outputs, and personally I doubt this is the case, however, it is possible that there is some, shall we say, corruption of the data here, since it was not possible to measure the ability of LLMs to predict the next word before there were LLMs.

I propose you do the same things, but only include HN content from before the existence of LLMs. That should ensure there is no bias towards any of the models.

Re: Are you better than a language model at predicting the next word?

#34

Post by JoelEinbinder »

DiscourseFan wrote: post #33
JoelEinbinder wrote: post #19

Earlier quoted context omitted.

The LLM didn’t generate the next word. Hacker News commenters did. You can see the source of the comment on the results screen.

At this point, we've all gotten quite used to the "style" of LLM outputs, and personally I doubt this is the case, however , it is possible that there is some, shall we say, corruption of the data here, since it was not possible to measure the ability of LLMs to predict the next word before there were LLMs . I propose you do the same things, but only include HN content from before the existence of LLMs. That should e…

If I used old comments then it's likely that the models will have trained on them. I haven't tested if that makes a difference though.

Re: Are you better than a language model at predicting the next word?

#36

Post by moralestapia »

>the quintessential language model task of predicting the next word?

Based on what? The whole test is flawed because of this. Even different LLMs would choose different answers and there's no objective argument to make for which one is the best.

Re: Are you better than a language model at predicting the next word?

#37

Post by jonahx »

JoelEinbinder wrote: post #2

I made a little game/quiz where you try to guess the next word in a bunch of Hacker News comments and compete against various language models. I used llama2 to generate three alternative completions for each comment creating a multiple choice question. For the local language models that you are competing against, I consider them having picked the answer with the lowest total perplexity of prompt + answer. I am able t…

"This exercise helped me to understand how language models work on a much deeper level."

I'd like to hear more on this.

Re: Are you better than a language model at predicting the next word?

#38

Post by sorokod »

moralestapia wrote: post #36

>the quintessential language model task of predicting the next word? Based on what? The whole test is flawed because of this. Even different LLMs would choose different answers and there's no objective argument to make for which one is the best.

The one provided in the original post.

Re: Are you better than a language model at predicting the next word?

#40

Post by e12e »

zoklet-enjoyer wrote: post #20

You scored 6/15. The best language model, gpt-4o, scored 6/15. The unigram model, which just picks the most common word without reading the prompt, scored 2/15. Keep in mind that you took 204 seconds to answer the questions, whereas the slowest language model was llama-3-8b taking only 10 seconds!

    you: 8/15
    gpt-4o: 2/15
    gpt-4: 4/15
    gpt-4o-mini: 4/15
    llama-2-7b: 5/15
    llama-3-8b: 5/15
    mistral-7b: 6/15
    unigram: 5/15
> You scored 8/15. The best language model, mistral-7b, scored 6/15. The unigram model, which just picks the most common word without reading the prompt, scored 5/15.

(In I think 120 seconds - didn't copy that part).

Interesting that results differ this much between runs (for the LLMs).

Surely someone did better than me on their first run?

Ed: I wonder if the human scores correlate with age of hn account?

Post reply on HN