81–90 of 105 posts
Where do the incorrect options come from?
In another comment the author wrote > I made a little game/quiz where you try to guess the next word in a bunch of Hacker News comments So I guess the correct answer comes from the HN user who wrote the comment?
I see that you get a random quiz every time, so results aren't comparable between people. I think I got an easy one. Neat game! If you could find a corpus that makes it easy for average humans to beat the LLMs, and add some nice design, maybe Wordle-style daily challenge plus social sharing etc, I could see it going viral just as a way for people to "prove" that they are "smarter" than AI.
> You scored 11/15. The best language model, llama-2-7b, scored 10/15. I see that you get a random quiz every time, so results aren't comparable between people. I think I got an easy one. Neat game! If you could find a corpus that makes it easy for average humans to beat the LLMs, and add some nice design, maybe Wordle-style daily challenge plus social sharing etc, I could see it going viral just as a way for people…
> You scored 28/100. The best language model, gpt-4, scored 32/100. The unigram model, which just picks the most common word without reading the prompt, scored 28/100.
Assuming complexity averages out on N=100, small test with LLM score above ~5 is "easy"
Earlier quoted context omitted.
They're perfectly capable of inviting you out for coffee. They just can't show up yet.
though, with web access and a credit card and the right information, you could probably get one to order a pizza to your house though.
This is a nonsense test. There is no context, so the 'next' word after the single word 'The' is effectively random. I'm pretty certain that LLMs are unable to work at all without context.
IMHO that doesn't make it nonsense, but maybe you are reading something different into the purpose of this test to what I am.
Nice. I found you can beat this by picking the word least likely to be selected by a language model, because it seems like the alternative choices are generated by an LLM. “Pick the outlier” is the best strategy. This is presumably also a simply strategy for detecting AI content in general - see how many “high temperature” choices it makes.