7/15, 90 seconds.
I'll blame it on fact that I'm not English native speaker, right? Right?
On a more serious note it was a cool thing to go through! It seemed like something that should have been so easy at first glance.
Are you better than a language model at predicting the next word?
41–50 of 105 posts
Re: Are you better than a language model at predicting the next word?
#42It's a neat idea, though not what I expected from the title talking about "smart" :) You might want to replace the single page format with showing just one question at a time, and giving instant feedback on after each answer. First, it'd be more engaging. Even the small version of the quiz is a bit long for something where you don't know what the payoff will be. Second, you'd get to see the correct answer while still…
> not what I expected from the title talking about "smart"
I think the title is mainly a reference to the TV show “Are you smarter than a fifth grader?”
Fittingly then, is the fact that a lot of types of questions that they were asking in that TV show was mostly trivia. Which I also don’t think of as being a particularly important characteristic of being “smart”.
When I think of “smart” people, I think of people who can take limited amount of information and connect dots in ways that others can’t. Of course it also builds on knowledge. You need to have specific knowledge in the first place to make connections. But knowing facts like “the battle of so and so happened on August 18th 1924, one hundred years ago today” alone is not “smart”. A smart person is someone who uses knowledge in a surprising way. Or in a way that others would not have been able to. After the smart person made the connection others might also go like “oh that’s so obvious why didn’t I think about that” or even “yeah that’s really obvious, I could’ve thought of that too”. And yet the first person to actually make, and properly communicate that connection was the smart one. Smart exactly because they did.
Re: Are you better than a language model at predicting the next word?
#43Earlier quoted context omitted.
The LLM didn’t generate the next word. Hacker News commenters did. You can see the source of the comment on the results screen.
At this point, we've all gotten quite used to the "style" of LLM outputs, and personally I doubt this is the case, however , it is possible that there is some, shall we say, corruption of the data here, since it was not possible to measure the ability of LLMs to predict the next word before there were LLMs . I propose you do the same things, but only include HN content from before the existence of LLMs. That should e…
an unbiased llm shouldn't be producing "style", it should be generating outputs that closely match the training set, as such their introduction should constitute only some biasing toward the average, which also happens in language usage in humans over time. the outcome is likely indistinguishable for large general data sets and large models. i am interested to see how chatbot outputs produce human output bias in generations growing up with them though, that seems likely and will probably be substantial
Re: Are you better than a language model at predicting the next word?
#44I made a little game/quiz where you try to guess the next word in a bunch of Hacker News comments and compete against various language models. I used llama2 to generate three alternative completions for each comment creating a multiple choice question. For the local language models that you are competing against, I consider them having picked the answer with the lowest total perplexity of prompt + answer. I am able t…
It's an interesting test, pretty cool idea. Thanks for sharing
Re: Are you better than a language model at predicting the next word?
#45Got 8/15, best AI model got 7/15, and unigram got 1/15. Finally a use for all the wasted hours I’ve spent on HN — my next word prediction is marginally better than that of the AI.
I have wasted an inordinate amount of time hn. i scored 2/15
Re: Are you better than a language model at predicting the next word?
#467/15, 90 seconds. I'll blame it on fact that I'm not English native speaker, right? Right? On a more serious note it was a cool thing to go through! It seemed like something that should have been so easy at first glance.
I am a native English speaker and only got 5/15 - and it took me over 100 seconds. You have permission to bask in the glory of your superiority over both GPT4 and your fellow HN readers!
Re: Are you better than a language model at predicting the next word?
#47It's a neat idea, though not what I expected from the title talking about "smart" :) You might want to replace the single page format with showing just one question at a time, and giving instant feedback on after each answer. First, it'd be more engaging. Even the small version of the quiz is a bit long for something where you don't know what the payoff will be. Second, you'd get to see the correct answer while still…
If you want to practice it one question at at time, you set the question count to 1. https://joel.tools/smarter/?questions=1 When I tested it this way it resulted in less of an emotional reaction.
I retired as worldwide champion (tied) of text prediction.
you: 0/1
gpt-4o: 0/1
gpt-4: 0/1
gpt-4o-mini: 0/1
llama-2-7b: 0/1
llama-3-8b: 0/1
mistral-7b: 0/1
unigram: 0/1Re: Are you better than a language model at predicting the next word?
#48I got one of my own comments on the 15 question quiz!
Re: Are you better than a language model at predicting the next word?
#49This isn't really the challenge (loss function) that language models are trained on. It's not a simple next-word challenge, they get more context, see how BERT was trained as a reference.
Re: Are you better than a language model at predicting the next word?
#50Where do the incorrect options come from?
I suspect they come from the LLMs.