Live data from Hacker News

Bag of words, have mercy on us

experimental-history.com

321–330 of 362 posts

Re: Bag of words, have mercy on us

#321

Earlier quoted context omitted.

I'm definitely a stream of words. My "abstract thoughts" are a stream of words too, they just don't get sounded out. Tbf I'd rather they weren't there in the first place. But bodies which refuse to harbor an "interiority" are fast-tracked to destruction because they can't suf^W^W^W be productive. Funny movie scene from somewhere. The sergeant is drilling the troops: "You, private! What do you live for!", and expects…

Then what are non-human animals doing?

Excellent point. See also the failure of Sapir-Whorf to prove that language determines thought. I think we have plenty of evidence that, while language can influence thought, it is not thought itself. Many people invested in AI are happy to throw out decades of linguistic evidence that language and thought are separate.

Re: Bag of words, have mercy on us

#322
post #265

Earlier quoted context omitted.

If you want actionable intuition, try "a human with almost zero self-awareness". "Self-awareness" used in a purely mechanical sense here: having actionable information about itself and its own capabilities. If you ask an old LLM whether it's able to count the Rs in "strawberry" successfully, it'll say "yes". And then you ask it to do so, and it'll say "2 Rs". It doesn't have the self-awareness to know the practical l…

Agree that's a better intuition, with pretraining pushing the model towards saying "I don't know" in the kinds of situations where people write that as opposed to by introspection of its own confidence.

There appears to be a degree of "introspection of its own confidence" in modern LLMs. They can identify their own hallucinations, at a rate significantly better than chance. So there must be some sort of "do I recall this?" mechanism built into them. Even if it's not exactly a reliable mechanism.

Anthropic has discovered that this is definitely the case for name recognition, and I suspect that names aren't the only things subject to a process like that.

Re: Bag of words, have mercy on us

#323
post #32
post #3

Every day I see people treat gen AI like a thinking human, Dijkstra's attitudes about anthropomorphizing computers is vindicated even more. That said, I think the author's use of "bag of words" here is a mistake. Not only does it have a real meaning in a similar area as LLMs, but I don't think the metaphor explains anything. Gen AI tricks laypeople into treating its token inferences as "thinking" because it is traine…

I'll make the following observation: The contra-positive of "All LLMs are not thinking like humans" is "No humans are thinking like LLMs" And I do not believe we actually understand human thinking well enough to make that assertion. Indeed, it is my deep suspicion that we will eventually achieve AGI not by totally abandoning today's LLMs for some other paradigm, but rather embedding them in a loop with the right pers…

Given that LLMs are incapable of synthetic a priori knowledge and humans are, I would say that as the tech stands currently, it's reasonable to make both of those statements.

Re: Bag of words, have mercy on us

#324
post #3

Every day I see people treat gen AI like a thinking human, Dijkstra's attitudes about anthropomorphizing computers is vindicated even more. That said, I think the author's use of "bag of words" here is a mistake. Not only does it have a real meaning in a similar area as LLMs, but I don't think the metaphor explains anything. Gen AI tricks laypeople into treating its token inferences as "thinking" because it is traine…

Bag of words is actually the perfect metaphor. The data structure is a bag. The output is a word. The selection strategy is opaquely undefined. > Gen AI tricks laypeople into treating its token inferences as "thinking" because it is trained to replicate the semiotic appearance of doing so. A "bag of words" doesn't sufficiently explain this behavior. Something about there being significant overlap between the smartest…

I think it's still a bit of a tortured metaphor. LLMs operate on tokens, not words. And to describe their behavior as pulling the right word out of a bag is so vague that it applies every bit as much to a Naive Bayes model written in Python in 10 minutes as it does to the greatest state of the art LLM.

Re: Bag of words, have mercy on us

#325
post #218

Earlier quoted context omitted.

I'm definitely a stream of words. My "abstract thoughts" are a stream of words too, they just don't get sounded out. Tbf I'd rather they weren't there in the first place. But bodies which refuse to harbor an "interiority" are fast-tracked to destruction because they can't suf^W^W^W be productive. Funny movie scene from somewhere. The sergeant is drilling the troops: "You, private! What do you live for!", and expects…

I doubt words are involved when we e.g. solve a mathematical problem. To me, solving problems happens in a logico/aesthetical space which may be the same as when you are intellectually affected by a work of art. I don't remember myself being able to translate directly into words what I feel for a great movie or piece of music, even if in the late I can translate this "complex mental entity" into words, exactly like I…

My wordmangling and mathsolving happen in that sort of logico/aesthetical space, too!

The twist about words in particular is they are distinctly articulable symbols, i.e. you can sound 'em out - and thus, presumably, have a reasonable expectation for bearers of the same language to comprehend if not what you meant then at least some vaguely predictable meaning-cloud associated with the given speech act.

That's unlike e.g. the numbers (which are more compressed, and thus easier to get wrong), or the syntagms of a programming language (which don't even have a canonical sonic representation).

Therefore, it's usually words that are taught to a mind during the formative stages of its emergence. That is, the words that you are taught, your means of inner reflection, are still sort of an imposition from the outside.

Just consider what you life trajectory would've been if in your childhood you had refused to learn any words, or learned them and then refused to mistake them for the things they represent!

Infants and even some animals recognize their reflection in a mirror; however, practically speaking, introspection is something that one needs to be taught: after recognizing your reflection you still need to be instructed what is to be done about it.

Unfortunately, introspection needing to be taught means that introspection can be taught wrongly.

As you can see with the archetypical case of "old and wise person does something completely stupid in response to communication via digital device", a common failure mode of how people are taught introspection (and, I figure, an intentional one!) is not being able to tell apart yourself from your self, i.e. not having an intuitive sense of where the boundary lies between perception and cognition, i.e. going through life without ever learning the difference between the "you" and the "words about you".

It's extremely common, and IMO an extremely factory-farming kind of tragic.

I say it must be extremely intentional as well, because the well-known practice of using "introspection modulators" to establish some sort of perceptual point of reference (such as where the interior logicoaeshtetical space ends and exterior causalityspace begins) very often ends up with the user in, well, a cage of some sort.

Re: Bag of words, have mercy on us

#326
I'm partial to the metaphor I made up:

They are search engines that can remix results.

I like this one because I think most modern folks have a usefully accurate model of what a search engine is in their heads, and also what "remixing" is, which adds up to a better metaphor than "human machine" or whatever.

Re: Bag of words, have mercy on us

#327
post #70

Earlier quoted context omitted.

Are you a stream of words or are your words the “simplistic” projection of your abstract thoughts? I don’t at all discount the importance of language in so many things, but the question that matters is whether statistical models of language can ever “learn” abstract thought, or become part of a system which uses them as a tool. My personal assessment is that LLMs can do neither.

Words are the "simplistic" projection of an LLM's abstract thoughts. An LLM has: words in its input plane, words in its output plane, and A LOT of cross-linked internals between the two. Those internals aren't "words" at all - and it's where most of the "action" happens. It's how LLMs can do things like translate from language to language, or recall knowledge they only encountered in English in the training data whil…

The "cross-linked internals" only go one direction and only one token at a time, slide window and repeat. The RL layer then picks which few sequences of words are best based on human feedback in a single step. Even "thinking" is just doing this in a loop with a "think" token. It is such a ridiculously simplistic model that it is vastly closer to an adder than a human brain.

Re: Bag of words, have mercy on us

#328
post #41

Everyone is out here acting like "predicting the next thing" is somehow fundamentally irrelevant to "human thinking" and it is simply not the case. What does it mean to say that we humans act with intent? It means that we have some expectation or prediction about how our actions will effect the next thing, and choose our actions based on how much we like that effect. The ability to predict is fundamental to our abili…

Predicting the next token is not at all the same thing as predicting the next action in a causal chain of actions. Not even close. One is model of language tokens, the other is a model of the physical world. You can come up with all sorts of predictions that can't be expressed cleanly in natural language. And plenty of things that parse cleanly from a language perspective but are unhinged in their description of empirical reality.

Re: Bag of words, have mercy on us

#329
"An AI is a bag that contains basically all words ever written, at least the ones that could be scraped off the internet or scanned out of a book."

The quantitative and qualitative difference between (a) "all words ever written" and (b) "ones that could be scraped off the internet or scanned out of book" easily exceeds the size of any LLM

Compared to (a), (b) is a tiny pouch, not even a bag

Opinions may differ on whether (b) is a representative sample of (a)

The words "scanned out of a book" would seem to be the most useful IMHO but the AI companies do not have enough words from those sources to produce useful general purpose LLMs

They have to add words "that could be scraped off the internet" which, let's be honest, is mostly garbage

Re: Bag of words, have mercy on us

#330

As a consequence of my profession, I understand how LLMs work under the hood. I also know that we data and tech folks will probably never win the battle over anthropomorphization. The average user of AI, nevermind folks who should know better, is so easily convinced that AI "knows," "thinks," "lies," "wants," "understands," etc. Add to this that all AI hosts push this perspective (and why not, it's the easiest white…

You may know the mechanics, but you don't know how LLMs "work" because no one really understands (yet, hopefully).
Post reply on HN