I think this is a good joke on nay-sayers. But if author is here, I would like a clarification if user is picking the next token or the next word? Cause if it is the latter, I think this test is invalid.
Are you better than a language model at predicting the next word?
21–30 of 105 posts
Re: Are you better than a language model at predicting the next word?
#22> 8. All of local politics in the muni I live in takes place in a forum like this, on Facebook[.]
The electeds in our muni post on it; I've gotten two different local laws done by posting there (and I'm working on a bigger third); I met someone whose campaign I funded and helped run who is now a local elected. It is crazy to think you can HN-effortpost your way to changing the laws of the place you live in but I'm telling you right now that you can.
This is a magical experience. I've done something similar in my university's CS department when I pointed out how the learning experience in the first programming course varies too much depending upon who the professor is.
I've never experienced this anywhere else. American politicians at all levels don't appear to be the least bit responsive to the needs and issues of anyone but the wealthy and powerful.
Re: Are you better than a language model at predicting the next word?
#23I feel like I recognise the comment about tensors from HN a few days ago, haha.
Re: Are you better than a language model at predicting the next word?
#24So... If I picked the same results, in the same timeframe... And I don't think glue should go on pizza... Does that mean LLMs are completely useless to me?
Re: Are you better than a language model at predicting the next word?
#25With some brief experimentation ChatGPT also fails this test.
It might make sense: any kind of fine-tuning of LLMs usually reduces generalization capabilities, and instruction-tuning is a kind of fine-tuning.
Re: Are you better than a language model at predicting the next word?
#26I think this is a good joke on nay-sayers. But if author is here, I would like a clarification if user is picking the next token or the next word? Cause if it is the latter, I think this test is invalid.
The language model generating the candidate answers generates tokens until a full word is produced. The language models picking their answer choose the completion that results in the lowest perplexity independent of the tokenization.
Re: Are you better than a language model at predicting the next word?
#27Where do the incorrect options come from?
Re: Are you better than a language model at predicting the next word?
#28Just proves why IQ tests are worthless.
Re: Are you better than a language model at predicting the next word?
#295/15, so the same as choosing the most common word. I think I did worse when the prompt is shorter. It just becomes a guessing game then and I find myself thinking more like a language model.
It says choosing the most common word was just 1/5 (and their best LLM was 4/15)
Re: Are you better than a language model at predicting the next word?
#30This is the best interactive website about LLMs at a meta level (so excluding prompt interfaces for actual AIs) that I've seen so far.
Quizzes can be magical.
Haven't seen any cooler new language-related interactive fun-project on the web since:
It would be great if the quiz included an intro or note about the training data, but as-is it also succeeds because it's obvious from the quiz prompts/questions that they're related to HN comments.
Sharing this with a general audience could spark funny discussions about bubbles and biases :)