I got 9/15, vs. 4/15 for an LLM. I assume these are lifted from HN? Seems like an indication I should spend less time here...
Are you better than a language model at predicting the next word?
71–80 of 105 posts
Re: Are you better than a language model at predicting the next word?
#72you: 4/15 gpt-4o: 0/15 gpt-4: 1/15 gpt-4o-mini: 2/15 llama-2-7b: 2/15 llama-3-8b: 3/15 mistral-7b: 4/15 unigram: 1/15 Seems like none of us is really better than flipping a coin, so I'd wager that you cannot accurately predict the next word with the given information. If one could instead sort the answers by likelihood and got scored based on how high one ranked the correct answer, things would probably look better t…
On the full set of 1000 questions, the language models are getting 30-35% correct. With patience, humans can do 40-50%.
The language models were prompted with the text + each candidate answer, and the one with the lowest perplexity was picked. I tried to avoid instruction tuned models wherever possible to avoid the "voice" problem.
Re: Are you better than a language model at predicting the next word?
#73The LLMs are better than me at knowing the finer probabilities of next words, and worse than me at guessing the points being made and reasoning about that.
Re: Are you better than a language model at predicting the next word?
#74Earlier quoted context omitted.
The LLM didn’t generate the next word. Hacker News commenters did. You can see the source of the comment on the results screen.
Do LLM's generate words on the fly or can they sort of "go back" and correct themselves? stackghost brought up a good point I didn't think about before
Beam search generates multiple potential completions and scores multiple tokens by likelihood, the picks the most likely after some threshold or length, which is close to a "go back and try again".
Re: Are you better than a language model at predicting the next word?
#75Re: Are you better than a language model at predicting the next word?
#76Earlier quoted context omitted.
I can also invite you out for a coffee and your LLM can’t do that either–yet.
They're perfectly capable of inviting you out for coffee. They just can't show up yet.
though, with web access and a credit card and the right information, you could probably get one to order a pizza to your house though.
Re: Are you better than a language model at predicting the next word?
#77If the samples came from HN, I wonder how likely it is that the text is already a part of a dataset (ie common crawl snapshot) so that the LLMs have already seen them?
edit: judging from the comments I saw, they were all quite recent, so I guess this isn't happening. Though I do know that ChatGPT can sometimes use a Bing search tool during chats, which can actually link to recently indexed text, but I highly doubt that the gpt4o-mini API model is doing that.
Re: Are you better than a language model at predicting the next word?
#78Everything I picked was grammatically correct, so I don't see the point. Is the point of a "language model" just to recall people's comments from the internet now?
Re: Are you better than a language model at predicting the next word?
#79This is a nonsense test. There is no context, so the 'next' word after the single word 'The' is effectively random.
I'm pretty certain that LLMs are unable to work at all without context.
Re: Are you better than a language model at predicting the next word?
#80Everything I picked was grammatically correct, so I don't see the point. Is the point of a "language model" just to recall people's comments from the internet now?
Always has been.