The only ones I got right were ones where I had read the actual HN comment…
Are you better than a language model at predicting the next word?
91–100 of 105 posts
Re: Are you better than a language model at predicting the next word?
#92you: 5/15
gpt-4o: 5/15
gpt-4: 5/15
gpt-4o-mini: 4/15
llama-2-7b: 7/15
llama-3-8b: 7/15
mistral-7b: 7/15
unigram: 4/15
Re: Are you better than a language model at predicting the next word?
#93Of course not, but that does not mean LLMs will lead to AGI. We might never build AGI in fact: https://www.lycee.ai/blog/why-no-agi-openai
Re: Are you better than a language model at predicting the next word?
#94Of course not, but that does not mean LLMs will lead to AGI. We might never build AGI in fact: https://www.lycee.ai/blog/why-no-agi-openai
That article, disapointingly, doesn't provide any arguments as to why we can't build AGI.
Re: Are you better than a language model at predicting the next word?
#95Like a ML model I would prefer being scored with cross entropy and not right/wrong. Like, I might guess wrong but it might not be that far off in likelihood.
Re: Are you better than a language model at predicting the next word?
#96you: 4/15 gpt-4o: 0/15 gpt-4: 1/15 gpt-4o-mini: 2/15 llama-2-7b: 2/15 llama-3-8b: 3/15 mistral-7b: 4/15 unigram: 1/15 Seems like none of us is really better than flipping a coin, so I'd wager that you cannot accurately predict the next word with the given information. If one could instead sort the answers by likelihood and got scored based on how high one ranked the correct answer, things would probably look better t…
On the full set of 1000 questions, the language models are getting 30-35% correct. With patience, humans can do 40-50%. The language models were prompted with the text + each candidate answer, and the one with the lowest perplexity was picked. I tried to avoid instruction tuned models wherever possible to avoid the "voice" problem.
i'm curious, how did you arrive at "40-50%" possible human performance?
the task of "predicting the next word" can be understood as either "correctly choosing the next word in the hidden context", or "predicting the likelihood of each possible word".
the quiz is evaluating against the former, but humans are still far from being able to express a percentile likelihood for each possibility.
i only consciously arrive at a vague feeling of confidence, rather than being able to weigh the prediction of each word with fractional precision.
one might say that LLMs have above human introspective ability in that regard.
Re: Are you better than a language model at predicting the next word?
#97For anyone else daring the full 100 question quiz: you need to get at least a third right to be considered better than guessing by traditional statistical standards. (You'd need more than half to be better than LLMs.)
Re: Are you better than a language model at predicting the next word?
#98Like a ML model I would prefer being scored with cross entropy and not right/wrong. Like, I might guess wrong but it might not be that far off in likelihood.
It is mitigating that we get so many questions, but I agree it's inefficient. As a human forecaster I also prefer being judged in part on my confidence in each of the alternatives.
Re: Are you better than a language model at predicting the next word?
#99you: 6/15 (336sec)
gpt-4o: 5/15
gpt-4: 5/15
gpt-4o-mini: 5/15
llama-2-7b: 6/15
llama-3-8b: 6/15 (Slowest Bot: 14sec)
mistral-7b: 5/15
unigram: 2/15
Re: Are you better than a language model at predicting the next word?
#100Earlier quoted context omitted.
I retired as worldwide champion (tied) of text prediction. you: 0/1 gpt-4o: 0/1 gpt-4: 0/1 gpt-4o-mini: 0/1 llama-2-7b: 0/1 llama-3-8b: 0/1 mistral-7b: 0/1 unigram: 0/1
Uhm I was just wondering if all models could get a question correct at the same time and except this "you" model all got it correct. you: 0/1 gpt-4o: 1/1 gpt-4: 1/1 gpt-4o-mini: 1/1 llama-2-7b: 1/1 llama-3-8b: 1/1 mistral-7b: 1/1 unigram: 1/1
I found the you model being exceptionally bad at this. Where can I see how many I got right?