Are you better than a language model at predicting the next word?

If you're looking for "knowledge" try https://d.erenrich.net/are-you-smarter-than-an-llm/index.htm...

7/10 This is more about set shattering than 'smarts'

LLMs are effectively DAGs, they literally have to unroll infinite possibilities in the absence of larger context into finite options.

You can unroll and cyclic graph into a dag, but you constrict the solution space.

Take the 'spoken': sentence:

"I never said she stole my money"

And say it multiple times with emphasis on each word and notice how the meaning changes.

That is text being a forgetful functor.

As you can describe PAC learning, or as compression, which is exactly equivalent to the finite set shattering above, you can assign probabilities to next tokans.

But that is existential quantification, limited based on your corpus based on pattern matching and finding.

I guess if "Smart" is defined as pattern matching and finding it would apply.

But this is exactly why there was a split between symbolic AI, which targeted universal quantification and statistical learning, which targets existential quantification.

Even if ML had never been invented, I would assume that there were mechanical methods to stack rank next tokens from a corpus.

This isn't a case of 'smarter', but just different. If that difference is meaningful depends on context.

I like it. It's a humorous reversal of the usual articles that boil down to "Look! I made the AI fail at something!"

This is just a test of how likely you are to generate the same word as the LLM. The LLM does not produce the "correct" next word as there are multiple correct words that fit grammatically and can be used to continue the sentence while maintaining context.

I don't see what this has to do with being "smarter" than anything. Example:

1. I see a business decision here. Arm cores have licensing fees attached to them. Arm is becoming ____

a) ether

b) a

c) the

d) more

But who's to say which is "correct"? Arm is becoming a household name. Arm is becoming the premier choice for new CPU architectures. Arm is becoming more valuable by the day. Any of b), c), or d) are equally good choices. What is there to be gained in divining which one the LLM would pick?

You scored 6/15. The best language model, gpt-4o, scored 6/15. The unigram model, which just picks the most common word without reading the prompt, scored 2/15.

Keep in mind that you took 204 seconds to answer the questions, whereas the slowest language model was llama-3-8b taking only 10 seconds!