Sparks of Artificial General Intelligence: Early Experiments with GPT-4

Post by DontchaKnowit » Sat, Mar 25, 2023, 5:05 PM UTC

pixl97 wrote: post #188

DontchaKnowit wrote: post #179

Earlier quoted context omitted.

so strange. I heard that saying for the first time yesterday on a youtube video comment section. Weird I've seen in it twice in less than 24 hours

The phenomenon you are referring to is called the "Baader-Meinhof phenomenon," also known as the "frequency illusion" or "recency illusion." It occurs when you learn something new, such as a word, a concept, or an idea, and suddenly start noticing it everywhere. This effect is believed to be the result of a combination of selective attention and confirmation bias, which make you more likely to notice and remember thi…

Post by nl » Mon, Mar 27, 2023, 6:12 AM UTC

golol wrote: post #236

nl wrote: post #217

Earlier quoted context omitted.

I don't think that's clear at all. https://arxiv.org/abs/2212.10559 shows a LLM is doing gradient descent on the context window at inference time. If it's learning relationships between concepts at runtime based on information in the context window then it seems about as useful to say it is a Markov chain as it is to say that a human is a Markov chain. Perhaps we are, but the "current state" is unmeasurably complex.

Well all the information it learns at runtime is encoded in the context window. I don't feel like {tokens}^ctxWindow is unmeasurably complex. I think one should see a transformer as a stochastic computer operating on its memory. If you modelled a computer as a stochastic process, would you taje the state space to consist of the most recent instruction, or instead the whole memory of the computer?