Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are you better than a language model at predicting the next word?

joel.tools

Post reply on HN
Print view View original

51–60 of 105 posts

Re: Are you better than a language model at predicting the next word?

#51

Post by DiscourseFan »

raggi wrote: post #43
DiscourseFan wrote: post #33

Earlier quoted context omitted.

At this point, we've all gotten quite used to the "style" of LLM outputs, and personally I doubt this is the case, however , it is possible that there is some, shall we say, corruption of the data here, since it was not possible to measure the ability of LLMs to predict the next word before there were LLMs . I propose you do the same things, but only include HN content from before the existence of LLMs. That should e…

an unbiased llm shouldn't be producing "style", it should be generating outputs that closely match the training set, as such their introduction should constitute only some biasing toward the average, which also happens in language usage in humans over time. the outcome is likely indistinguishable for large general data sets and large models. i am interested to see how chatbot outputs produce human output bias in gene…

But that's clearly not the case. There was a post the other day about how GPT used certain words at a rate remarkably higher than average. Also the paragraph breaks, the politesse. No, I don't have much to back it up, but generally I can tell very quickly if a chunk of text is from ChatGPT, for instance, or if an image is generated by DALL-E.

Re: Are you better than a language model at predicting the next word?

#52

Post by raggi »

DiscourseFan wrote: post #51
raggi wrote: post #43

Earlier quoted context omitted.

an unbiased llm shouldn't be producing "style", it should be generating outputs that closely match the training set, as such their introduction should constitute only some biasing toward the average, which also happens in language usage in humans over time. the outcome is likely indistinguishable for large general data sets and large models. i am interested to see how chatbot outputs produce human output bias in gene…

But that's clearly not the case. There was a post the other day about how GPT used certain words at a rate remarkably higher than average. Also the paragraph breaks, the politesse. No, I don't have much to back it up, but generally I can tell very quickly if a chunk of text is from ChatGPT, for instance, or if an image is generated by DALL-E.

in the above, when i say llm, i mean the base models, when i say chatbot, i mean things like chatgpt, they're not the same. chatgpt is not just a frontend for the base model, studies on chatgpt covering output biasing that it has from the fine tuning, prompts and contexts and other things they do are largely not applicable to the raw model generation in this quiz, and they are also largely not applicable to llms as a whole

Re: Are you better than a language model at predicting the next word?

#53

Post by DiscourseFan »

raggi wrote: post #52
DiscourseFan wrote: post #51

Earlier quoted context omitted.

But that's clearly not the case. There was a post the other day about how GPT used certain words at a rate remarkably higher than average. Also the paragraph breaks, the politesse. No, I don't have much to back it up, but generally I can tell very quickly if a chunk of text is from ChatGPT, for instance, or if an image is generated by DALL-E.

in the above, when i say llm, i mean the base models, when i say chatbot, i mean things like chatgpt, they're not the same. chatgpt is not just a frontend for the base model, studies on chatgpt covering output biasing that it has from the fine tuning, prompts and contexts and other things they do are largely not applicable to the raw model generation in this quiz, and they are also largely not applicable to llms as a…

An LLM takes a slice of data from the world, by nature it has to organize it in some such way, depending on how its trained, and the method of organizing it is hard-coded into the model. Therefore, all models will develop some sort of style, no matter what, since somebody, or a team of people, had to figure out a way to portion out a selection of data, and this problem is intractable.

Re: Are you better than a language model at predicting the next word?

#54

Post by moralestapia »

sorokod wrote: post #38
moralestapia wrote: post #36

>the quintessential language model task of predicting the next word? Based on what? The whole test is flawed because of this. Even different LLMs would choose different answers and there's no objective argument to make for which one is the best.

The one provided in the original post.

I don't see any of that.

Quote?

Re: Are you better than a language model at predicting the next word?

#55

Post by dang »

jsnell wrote: post #5

It's a neat idea, though not what I expected from the title talking about "smart" :) You might want to replace the single page format with showing just one question at a time, and giving instant feedback on after each answer. First, it'd be more engaging. Even the small version of the quiz is a bit long for something where you don't know what the payoff will be. Second, you'd get to see the correct answer while still…

Thanks - we've LLMified the title.

Re: Are you better than a language model at predicting the next word?

#57

Post by YZF »

sigbottle wrote: post #31
JoelEinbinder wrote: post #19

Earlier quoted context omitted.

The LLM didn’t generate the next word. Hacker News commenters did. You can see the source of the comment on the results screen.

Do LLM's generate words on the fly or can they sort of "go back" and correct themselves? stackghost brought up a good point I didn't think about before

afaik they do not go back. keep in mind there is a context in which they are generating the response, e.g. the system prompt and the actual question.

Re: Are you better than a language model at predicting the next word?

#58

Post by JoelEinbinder »

rlt wrote: post #56

Is this with the “temperature” parameter set to 0? Most LLM chatbots set it to something higher. It would be interesting to try varying it, as well as the seed.

Temperature doesn't play a role here, because the LLM is not being sampled (other than to generate the candidate answers). Instead the answer the llm picks is decided by computing the complexity for the full prompt + answer string.

Re: Are you better than a language model at predicting the next word?

#59

Post by JoelEinbinder »

moralestapia wrote: post #54
sorokod wrote: post #38

Earlier quoted context omitted.

The one provided in the original post.

I don't see any of that. Quote?

The prompts you see in the quiz are from real hacker news comments. Whatever word the commenter said next is the "correct" word.

Re: Are you better than a language model at predicting the next word?

#60

Post by moralestapia »

JoelEinbinder wrote: post #59
moralestapia wrote: post #54

Earlier quoted context omitted.

I don't see any of that. Quote?

The prompts you see in the quiz are from real hacker news comments. Whatever word the commenter said next is the "correct" word.

This is what I see,

  Are you smarter than a language model?

  There are a lot of benchmarks that try to see how good language models are at human tasks. But how good are you at the quintessential language model task of predicting the next word?
And then a list of questions.

How am I supposed to know it has anything to do with HN?

Post reply on HN