Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude's new system prompt doesn't want to reproduce song lyrics

simonwillison.net

Post reply on HN
Print view View original

121–127 of 127 posts

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#121

Post by qarl »

GPerson wrote: post #117
qarl wrote: post #114

Earlier quoted context omitted.

> It just seems absurd to me. But this is just the "consciousness is based on information, not substrate" position, right? It seems absurd? I understand that it troubles you that real computation can be done with marbles... but... it can. So you must believe that consciousness is entirely uncomputable? Do you have any other examples of physical processes that are beyond computation? > A priori why must real concepts…

> So you must believe that consciousness is entirely uncomputable? Do you have any other examples of physical processes that are beyond computation? I think all physical processes are distinct from their descriptions, and have a real existence. We can describe some things about them but our descriptions are not equal to them.

Hey - I noticed you're not replying to me. No worries - I understand that these conversations can be a bit much.

I just wanted to thank you - our talk helped me develop my ideas around "emotion faking" - which I was having trouble articulating.

Thank you.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#122

Post by GPerson »

qarl wrote: post #120
GPerson wrote: post #118

Earlier quoted context omitted.

Let me apologize for having to discontinue now, or my girlfriend will dump me for being more interested in talking to you than her tonight. Another time!

OMG. Yes, of course. Send your girlfriend my regards. I didn't see your second reply above - so I answered it now. I look forward to continuing... tomorrow?

Sorry it’s been two days. I’ll try to remember where we left off and pick up later.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#123

Post by GPerson »

qarl wrote: post #121
GPerson wrote: post #117

Earlier quoted context omitted.

> So you must believe that consciousness is entirely uncomputable? Do you have any other examples of physical processes that are beyond computation? I think all physical processes are distinct from their descriptions, and have a real existence. We can describe some things about them but our descriptions are not equal to them.

Hey - I noticed you're not replying to me. No worries - I understand that these conversations can be a bit much. I just wanted to thank you - our talk helped me develop my ideas around "emotion faking" - which I was having trouble articulating. Thank you.

Oh whoops I replied above apologizing for my delay. I’m starting a new job and they’re threatening to make me do stuff (a great thing!!) But I will likely send my responses tonight.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#124

Post by GPerson »

qarl wrote: post #119
GPerson wrote: post #117

Earlier quoted context omitted.

> So you must believe that consciousness is entirely uncomputable? Do you have any other examples of physical processes that are beyond computation? I think all physical processes are distinct from their descriptions, and have a real existence. We can describe some things about them but our descriptions are not equal to them.

I'm sorry - I missed this reply - which is why I asked it a second time below. So - let me know if I have this right - if I created a simulation of you it would act like you and talk like you in all ways - but not actually feel like you? And if I asked it if it felt like you (being a perfect simulation of you) it would answer as you do "Yes, I feel like me." But it would be what... lying about having feelings? Do I h…

I thought about how to respond to this for quite a while tonight (last night now as I fell asleep).

My position is that there must be something wrong with the idea that a behaviorally identical simulation can be made which is not conscious. I think this because I do think phenomenal consciousness is playing a role in behavior, since surely my consciousness is a cause of memories of my consciousness (certain structure of physical matter in my brain has been caused by my consciousness); this is called the self-stultification argument against epiphenomenalism.

So that just reaffirms my position, it does not argue against yours. I believe our physical models of reality are fundamentally limited and even within the limitations far from complete.

A truly behaviorally accurate simulation, by which I mean the simulation has parts corresponding to everything in the original, is a very high bar of course. So to address your point about faking directly, I would not call this faking, since faking seems to require a separate inner process to decide to do the deception, but I believe it is just a highly accurate mimic, and I don’t actually see anything wrong with this.

This message is getting long.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#125

Post by GPerson »

qarl wrote: post #116
GPerson wrote: post #115

Earlier quoted context omitted.

> I understand that it troubles you that real computation can be done with marbles... but... it can. I don’t understand what you’re saying here. It doesn’t trouble me at all. I’m actually quite confident I could design a simple computer from scratch in a substrate of marbles and an electric motor to bring them back up. Maybe we should separate these threads otherwise it’ll be too hard to keep branching out.

I must have misunderstood where you were drawing the line of absurdity. I'm comfortable going slower - although the site won't let me reply as quickly as it let's you. So I take it that you disbelieve that consciousness/qualia can be created by computation. But you do believe, I suspect, that consciousness/qualia are a physical process. So I'm wondering, is this the only example of non-computable physical process you…

I’ll just leave this comment to start a thread to ask you about your position and then figure out how I’d like to address your question later.

Your position seems to have non-trivial difficulties if its own. I take it you don’t believe that a sequence of unique rock states or wall states is conscious, which means consciousness is not merely caused by an abstract mapping to some sequence of information states. I.e., I gather you must believe there is something about the organization or causal structure of the processing necessary for consciousness. Before continuing I’ll share another thought experiment just to suggest why this must be. Put two supposed conscious AIs into conversation for a day. Take a long time writing down the states line by line in a (very!) big notebook. Then cover up each line with other sheets who paper, revealing them line by line in a sequence. From our perspective the same conversation must be getting expressed by the two AIs, but surely we don’t think this results in a new conscious experience for the two AIs modeled by the states.

Okay then my challenge to your position (and mine of course, but this is why I’m led to my beliefs ultimately) is it must explain (1) which kinds of casual and organizational structures count here and (2) how these result in conscious experience.

It seems to me that even though I also cannot meaningfully answer these, believing there is some special physics going on which is yet to be discovered seems necessary. I.e. the how part is that it’s just a feature of that physical process. The which kinds are “the physical ones of the right kind inherently have consciousness”. And I will point out that there has rarely been a time in the last 2000 years when we as humans did not think we were nearly at the end of understanding physics.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#126

Post by qarl »

GPerson wrote: post #124
qarl wrote: post #119

Earlier quoted context omitted.

I'm sorry - I missed this reply - which is why I asked it a second time below. So - let me know if I have this right - if I created a simulation of you it would act like you and talk like you in all ways - but not actually feel like you? And if I asked it if it felt like you (being a perfect simulation of you) it would answer as you do "Yes, I feel like me." But it would be what... lying about having feelings? Do I h…

I thought about how to respond to this for quite a while tonight (last night now as I fell asleep). My position is that there must be something wrong with the idea that a behaviorally identical simulation can be made which is not conscious. I think this because I do think phenomenal consciousness is playing a role in behavior, since surely my consciousness is a cause of memories of my consciousness (certain structure…

Thank you for continuing to engage.

> A truly behaviorally accurate simulation, by which I mean the simulation has parts corresponding to everything in the original, is a very high bar of course. So to address your point about faking directly, I would not call this faking, since faking seems to require a separate inner process to decide to do the deception, but I believe it is just a highly accurate mimic

Yes, this is the situation which is most interesting, despite the bar.

Ok - we can call it mimicry if you like. That doesn't change my question -

If the simulation isn't using the same mechanism (feelings) as the physical system, then where is this mimicry coming from? Clearly the simulation is just following the laws of chemistry. No mimicry in there. Is it in the original atomic configuration - that seems unlikely.

You seem to be assuming the existence of some mechanism that was not described in the original physical system. Why do you insist that this must be the case, and is not simply the same as the physical system?

I feel like you're on the wrong side of Occam. There is an obvious answer to this question - and yet you're picking the hypothetical/undiscovered/possibly non-existent one.

(Remember, in this situation we are not describing an LLM, but a system generated from the rules of chemistry. I can see how mimicry might apply in the LLM case - but I don't see where it would be hiding in the physical simulation case.)

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#127

Post by GPerson »

qarl wrote: post #126
GPerson wrote: post #124

Earlier quoted context omitted.

I thought about how to respond to this for quite a while tonight (last night now as I fell asleep). My position is that there must be something wrong with the idea that a behaviorally identical simulation can be made which is not conscious. I think this because I do think phenomenal consciousness is playing a role in behavior, since surely my consciousness is a cause of memories of my consciousness (certain structure…

Thank you for continuing to engage. > A truly behaviorally accurate simulation, by which I mean the simulation has parts corresponding to everything in the original, is a very high bar of course. So to address your point about faking directly, I would not call this faking, since faking seems to require a separate inner process to decide to do the deception, but I believe it is just a highly accurate mimic Yes, this i…

> Remember, in this situation we are not describing an LLM, but a system generated from the rules of chemistry

However, it is a system with a representation purported to be a correct description of all the relevant rules of chemistry and physics, and we don’t have knowledge of what all the relevant rules are.

And I might add that there is an assumption here that the relevant rules are representable via arbitrary description.

There is a basic idea I hold which we can highlight now. Since you find it implausible that there is such a description as alluded above beyond the currently known theories, you are led to conclude that there must not be anything else in reality. I don’t see why we can assume a priori that every true aspect of reality is arbitrarily representable independently of the substrate. In fact qualia and experience seem to be a first person view of processes which have third person effects. It seems like any precise description of this system could only refer to the third person effects, not the first person effects (since the description is third person).

> I feel like you're on the wrong side of Occam. There is an obvious answer to this question - and yet you're picking the hypothetical/undiscovered/possibly non-existent one.

I don’t think Occam’s razor or questioning my biases etc.. have any explanatory value for the actual question at hand. They might have value for ethics or sociology, but I want to know the actual truth. The substrate independent view which somehow conjures consciousness just seems less plausible than the alternative to me.

Post reply on HN