Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

They’re made out of weights

maxleiter.com

Post reply on HN
Print view View original

191–200 of 739 posts

Re: They’re made out of weights

#191

Post by bayindirh »

coldtea wrote: post #188
bayindirh wrote: post #180

Earlier quoted context omitted.

> Not sure which rhetorical trick is that. The point of the story, as I read it, is the technical insight (and some social implications of it). Take a simple mechanism which has exceedingly low number of inputs and states and create a narrative around it to convey it as intelligent. For a toaster, I can rewrite the think as "They're made of metal strips!", pointing out that their thermostat is a bimetal strip, and ex…

> Take a simple mechanism which has exceedingly low number of inputs and states and create a narrative around it to convey it as intelligent. (...) For a toaster, I can rewrite the think as "They're made of metal strips!", pointing out that their thermostat is a bimetal strip, and extrapolate from there. Doesn't that miss the whole point? You could write "They're made of metal strips!". You wouldn't be able to write…

Oh, no. When you add thermal sensing and offsetting ambient temperature, you can add all kind of "seemingly" intelligent features like feeling the emotions of the bread, and creating the perfect toast without hurting the cute little bread slices, making them perfectly blonde while not showing cruelty to them.

They can even adapt to their environment and the characteristics of the bread even with simplest of mechanisms because the text will be overglossing the fact that different types of breads have different thermal characteristics and this will deeply affect the behavior of the metal strips, bordering near a sentient being even more thoughtful and considerate than a human which is rushing through house to catch the bus in the morning.

Re: They’re made out of weights

#192

Post by slopinthebag »

eszed wrote: post #187
doctoboggan wrote: post #51

Earlier quoted context omitted.

> This is not meant as a gotcha, I am genuinely curious how you believe consciousness can be an emergent property. I was about to post the exact opposite question? How could it not be an emergent property? Unlike consciousness, the concept of emergence is pretty well defined: An emergent property is a characteristic or behavior that a complex system has, but which its individual components do not have on their own. C…

I'd expand a little bit to say inevitably emergent property. That is to say, if you create a sufficiently complex information-processing network, some level of consciousness will result. With regards to current AI, we're a fair way away from building something with enough connections, but we'll get there. One thing that gives me pause about the inevitability hypothesis is that the type of connection, or manner of inf…

> That is to say, if you create a sufficiently complex information-processing network, some level of consciousness will result.

Why is that?

Re: They’re made out of weights

#194

Post by aureate »

Assume LLMs have conscious experiences. Take a session with an LLM. A prompt is fed to the LLM. It generates some text. Another input is fed in, comprising the previous prompt, the generated text and a new prompt. The model generates some more text. This continues for a while and the session concludes.

Some questions:

1. Let's say we perform the exact same experiment, running the same program on the same computer with the same inputs and the same random seed. The same outputs are produced. The session is byte for byte identical in all the inputs, outputs and internal states. Is the conscious experience of the LLM here the same? If so, in what sense is it the same? Is it a similarity of two separate experiences or is it the same actual experience?

2. Now let's say the program that runs this LLM is rewritten from scratch and run on a different machine. The software and hardware are different but the weights are the same and all the inference calculations produce identical numbers. Is the conscious experience the same? In which sense?

3. Now say the weights are changed but the tokens generated for this particular session don't change. Same conscious experience?

4. Lastly, consider the original experiment. Did the LLM have a conscious experience corresponding to that first prompt and its response? Was that distinct from its conscious experience of the second prompt? Was the first experience then re-experienced every time the first prompt was fed back in as part of the later prompting steps? If so, what about the text of its own that it previously generated and is now fed back into it. Does this generate a conscious experience of its own?

And a further question - a dichotomy:

A. If the answer to 1 above is that the conscious experience is the same in the true identity sense - i.e. only one conscious experience is had, not a separate one in each run, does that imply that the conscious experience exists independently of any particular realisation of this experiment? If running this experiment N times results in exactly 1 conscious experience, is that still true if N=0?

B. On the other hand, if the two experiences are distinct (however similar they may be), how does that fit with the answer to question 4? A single consciousness experiencing the whole conversation in question 4 would seem at odds with the conscious experiences in question 1 being distinct, so doesn't this imply there is no conscious experience of the whole "conversation", but rather a separate conscious experience of each round of feed-all-the-prompts-and-outputs-back-in?

My own response to all of the above is "mu" - unask the question. It is ill-posed, sound-of-one-hand-clapping stuff. I think the questions assume properties that conscious experience simply doesn't have (particularly, the ability to perfectly reproduce the circumstances in which they arise), and that the questions simply don't make any sense in relation to actual conscious experience.

However, that way of thinking follows from a particular world view that many here don't share. I'm curious what thoughts people who take seriously the idea of LLM (or algorithmic, in general) consciousness have on the above questions.

Re: They’re made out of weights

#195

Post by noosphr »

throw310822 wrote: post #171
noosphr wrote: post #54

Earlier quoted context omitted.

As opposed to integers in normal programming. The 'magic' in weights is that the rules are spread through the whole model and you can't point to one place which encodes them. The grokking paper shows that this stops being the case with enough training data and enough compute.

Integers in normal programming represent data or instructions; instructions are hand coded, have rigidly defined semantics, are not differentiable and have no redundancy. > The 'magic' in weights is that the rules are spread through the whole model ... The grokking paper shows that this stops being the case with enough training data and enough compute. I don't understand what you mean to say. That weights are not mag…

>Integers in normal programming represent data or instructions; instructions are hand coded, have rigidly defined semantics, are not differentiable and have no redundancy.

I can, and have, written programs using an evolutionary algorithm that then run on bare metal. None of the things you list are true for those programs, yet other than being computationally more expensive to train they work just as well as neural networks.

>I don't understand what you mean to say

The diffusness of weights across the whole model isn't an innate feature of deep learning models. It is a feature of sparse training data and little compute.

Re: They’re made out of weights

#196

Post by sumitkumar »

akie wrote: post #168
sumitkumar wrote: post #148

The weights start with a random manifold. The training takes data and shapes the manifold, weight by weight, in many cycles. Once the training is the done manifold is fixed. When a new inference has to be done the query(q) is projected in the manifold space. This projection is dropped on the manifold and the gravity of the manifold gives an answer of q+1 length. Which(qw+i) is dropped qw+n times to output a final res…

That's a very concise and illuminating way to think about what's happening, IF (and only if) you already know how these models work. Thanks for that.

Yes this is more like compression to remember and not for learning/understanding.

Re: They’re made out of weights

#197

Post by DougBTX »

sumitkumar wrote: post #148

The weights start with a random manifold. The training takes data and shapes the manifold, weight by weight, in many cycles. Once the training is the done manifold is fixed. When a new inference has to be done the query(q) is projected in the manifold space. This projection is dropped on the manifold and the gravity of the manifold gives an answer of q+1 length. Which(qw+i) is dropped qw+n times to output a final res…

The weights are code, the prompt is code, the output is code.

Is the meat code?

Re: They’re made out of weights

#198

Post by maxbond »

bayindirh wrote: post #191
coldtea wrote: post #188

Earlier quoted context omitted.

> Take a simple mechanism which has exceedingly low number of inputs and states and create a narrative around it to convey it as intelligent. (...) For a toaster, I can rewrite the think as "They're made of metal strips!", pointing out that their thermostat is a bimetal strip, and extrapolate from there. Doesn't that miss the whole point? You could write "They're made of metal strips!". You wouldn't be able to write…

Oh, no. When you add thermal sensing and offsetting ambient temperature, you can add all kind of "seemingly" intelligent features like feeling the emotions of the bread, and creating the perfect toast without hurting the cute little bread slices, making them perfectly blonde while not showing cruelty to them. They can even adapt to their environment and the characteristics of the bread even with simplest of mechanism…

Who have you ever heard make a sincere, good faith argument that a toaster is conscious? Who do you imagine would argue against you if you asserted they were mere automata?

If you can't identify anyone, then this analogy doesn't work.

Re: They’re made out of weights

#199

Post by noduerme »

lxgr wrote: post #179
noduerme wrote: post #162

Earlier quoted context omitted.

In what way is that different from any other model of reality that you'd use to winnow a dataset into an answer to a question? The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with. It's almost like people desperately want to give up their agency and creativi…

> beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with It’s less about being too tired and more about being realistic about the limits of understanding. Consider mass and energy flows in planet-scale systems: At some point we call these “weather” and change the tools with which we study them, but…

When we attempt to recreate those complex, planetary atmospheric phenomena in a box, we're doing so in order to measure and study them.

Making random turbulence in a box until it resembles the outside world, and calling it weather and extrapolating some predictive meaning from the result, is the total antithesis of what you're describing about why we come up with simplified models for impossibly complex systems. The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly. [Whereas] The purpose of LLM models is to give the illusion of answering questions while never answering why the answer was given. The difference is the difference between a scientist and a tarot card reader, an equation and an oracle.

People have a well known tendency to gravitate toward the shamanistic, oracular, and superstitous. Listen, I ran a casino for 6 years, I know. The impossibility of knowing how 80 layers of matrix multiplication led to a particular answer is in itself a psychological factor in choosing whether to accept the answer or to question it. People tend to err on the side of the over, in sports betting terms... or on the lazy side in general... and they will make whatever excuses they need to after the fact to justify their decisions. So now we have a machine that can act like an oracle and which you can also blame, but the blame goes into a void because this machine is stateless and is only a reflection of information, not an intentional refinery of data.

Sit next to a bank of slot machines for an hour and listen to the absolutely ridiculous shit most people will come up with to explain how they "know" if a machine is going to pay out soon, and then tell me if you think it's a good idea to give them an LLM in their pocket to answer their questions in whatever way they frame them.

Re: They’re made out of weights

#200

Post by InsideOutSanta »

bulbar wrote: post #55
doctoboggan wrote: post #51

Earlier quoted context omitted.

> This is not meant as a gotcha, I am genuinely curious how you believe consciousness can be an emergent property. I was about to post the exact opposite question? How could it not be an emergent property? Unlike consciousness, the concept of emergence is pretty well defined: An emergent property is a characteristic or behavior that a complex system has, but which its individual components do not have on their own. C…

I think the alternative is that our brain, somehow, is connected to some metaphysical aspect of reality which is what most religions believe.

The counter to that is that altering the brain directly alters the consciousness. I can take LSD and I literally change. I can have parts of my brain removed and parts of my self disappear. It's not like cutting off a leg, where I lose capabilities but am still the same me.

The logical conclusion is that the brain makes me.

Post reply on HN