61–70 of 105 posts
Where do the incorrect options come from?
> I made a little game/quiz where you try to guess the next word in a bunch of Hacker News comments
So I guess the correct answer comes from the HN user who wrote the comment?
Yes. I can tell you about things that happened this morning. Your language model cannot.
This is presumably also a simply strategy for detecting AI content in general - see how many “high temperature” choices it makes.
Earlier quoted context omitted.
The prompts you see in the quiz are from real hacker news comments. Whatever word the commenter said next is the "correct" word.
This is what I see, Are you smarter than a language model? There are a lot of benchmarks that try to see how good language models are at human tasks. But how good are you at the quintessential language model task of predicting the next word? And then a list of questions. How am I supposed to know it has anything to do with HN?
Earlier quoted context omitted.
This is what I see, Are you smarter than a language model? There are a lot of benchmarks that try to see how good language models are at human tasks. But how good are you at the quintessential language model task of predicting the next word? And then a list of questions. How am I supposed to know it has anything to do with HN?
After the quiz, the source is linked along with the full comment.
Nice. I found you can beat this by picking the word least likely to be selected by a language model, because it seems like the alternative choices are generated by an LLM. “Pick the outlier” is the best strategy. This is presumably also a simply strategy for detecting AI content in general - see how many “high temperature” choices it makes.
For the longer comments I understand, but for the ones where it's 1 or 2 words and many of the options are correct English phrases, I don't understand why there's bias towards one? Wouldn't we need a prompt here?
Also, I got bored halfway through and selected "D" for all of them
Earlier quoted context omitted.
If you want to practice it one question at at time, you set the question count to 1. https://joel.tools/smarter/?questions=1 When I tested it this way it resulted in less of an emotional reaction.
I retired as worldwide champion (tied) of text prediction. you: 0/1 gpt-4o: 0/1 gpt-4: 0/1 gpt-4o-mini: 0/1 llama-2-7b: 0/1 llama-3-8b: 0/1 mistral-7b: 0/1 unigram: 0/1
you: 0/1
gpt-4o: 1/1
gpt-4: 1/1
gpt-4o-mini: 1/1
llama-2-7b: 1/1
llama-3-8b: 1/1
mistral-7b: 1/1
unigram: 1/1
It's a neat idea, though not what I expected from the title talking about "smart" :) You might want to replace the single page format with showing just one question at a time, and giving instant feedback on after each answer. First, it'd be more engaging. Even the small version of the quiz is a bit long for something where you don't know what the payoff will be. Second, you'd get to see the correct answer while still…
If you're looking for "knowledge" try https://d.erenrich.net/are-you-smarter-than-an-llm/index.htm...
I bet this could be a unique testing resource for aspiring Jeapordy contestants.
you: 4/15
gpt-4o: 0/15
gpt-4: 1/15
gpt-4o-mini: 2/15
llama-2-7b: 2/15
llama-3-8b: 3/15
mistral-7b: 4/15
unigram: 1/15
If one could instead sort the answers by likelihood and got scored based on how high one ranked the correct answer, things would probably look better than random.
Also I wonder how these LLMs were prompted. Were they just used to complete the text, or where they put in a "mood" where they would try to complete the text in the original author's voice?
Obviously as as human I'd try to put myself in the author's head and emulate their way of speaking, whereas an LLM might just complete things in its default voice.