Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude's new system prompt doesn't want to reproduce song lyrics

simonwillison.net

Post reply on HN
Print view View original

101–110 of 120 posts

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#101

Post by GPerson »

qarl2 wrote: post #88
simonw wrote: post #57

Earlier quoted context omitted.

Anthropic are uncomfortably interested in "model welfare" in my opinion - it's a regular feature of their system cards. Here's the Fable 5.1 PDF: https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32... - scroll to page 139.

Unless you actually believe there is something magical about the human spirit - someday we are going to make sentient machines. What exactly is wrong with preparing for that ahead of time? Just in case we pass that threshold before we realize it? Why does it make you uncomfortable?

There is a distinction between two important concepts here. There is access consciousness and there is phenomenal consciousness. The actual paper published alongside the j-space blog post discusses the idea of access consciousness if you are interested.

Phenomenal consciousness is what the hard problem of consciousness is about. I personally believe there is a good reason to think the reducibility of this concept to some substrate independent description is equivalent to having a proper definition, and also believe it is not reducible to such a description. Hence the only definition one can give is akin to pointing at the moon and hoping you don’t look at the hand. I.e., phenomenal consciousness is first person experience, the existence of qualia, and that the “lights are on”.

We are biased as educated people to believe all concepts are reducible to substrate independent description but there is no reason to believe this a priori.

The substrate non-independence of consciousness, if correct, could mean we won’t just build one no matter how intelligent and human-like they become.

I agree with you that there is nothing “magical” about our intelligence. However we have as a species effectively believed in magic since Newton decided gravity is a black box that cannot be reduced to the mechanical philosophy, and went on with describing its properties formally. We don’t call it magic, but physics. We call quantum mechanics unintuitive because it disagrees with the Boolean logic that is appear at the every day level. For some reason we keep making this mistake over and over.

(There is one weakness to my view that I aware of. I think there must have at least at some point and maybe still been a role in which the process that results in phenomenal conscious plays in a correlated intelligence, because it seems unlikely to persist through evolution if not. I.e. certain physical processes encode information better than others, and we happen to live in a universe where such process is aligned with consciousness. Substrate independence doesn’t have to contend with this, but it does have worse problems in my view.)

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#102

Post by qarl »

GPerson wrote: post #101
qarl2 wrote: post #88

Earlier quoted context omitted.

Unless you actually believe there is something magical about the human spirit - someday we are going to make sentient machines. What exactly is wrong with preparing for that ahead of time? Just in case we pass that threshold before we realize it? Why does it make you uncomfortable?

There is a distinction between two important concepts here. There is access consciousness and there is phenomenal consciousness . The actual paper published alongside the j-space blog post discusses the idea of access consciousness if you are interested. Phenomenal consciousness is what the hard problem of consciousness is about. I personally believe there is a good reason to think the reducibility of this concept to…

Yikes. You're going to have to be more succinct.

From my perspective, many of these arguments have the same structure as the arguments against evolution. Long, complex, and unintelligible.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#103

Post by GPerson »

simonw wrote: post #70
GPerson wrote: post #65

Earlier quoted context omitted.

You don’t act like you don’t want AGI. You have a very strong track record of speaking out of both sides of your mouth in different settings.

I've been pretty consistent about my disinterest in AGI: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... "You have a very strong track record of speaking out of both sides of your mouth in different settings." If I have a consistent record of that it should be very easy for you to come up with the some examples.

I was considering our interchange yesterday. I apologize for my rude manner in expressing my views. I do appreciate that you have the conscientiousness to engage with your detractors. From my perspective the whole AI Revolution has been deranged in many ways and it has perhaps deranged me as well. I do hold passionate views at odds with some of yours (and many others on this website). It does feel like an existential threat, so I don’t know if respectful calmness is appropriate in all circumstances. One must fight for your beliefs at the end of the day. But I’ll think about channeling my rage differently.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#104

Post by GPerson »

qarl wrote: post #102
GPerson wrote: post #101

Earlier quoted context omitted.

There is a distinction between two important concepts here. There is access consciousness and there is phenomenal consciousness . The actual paper published alongside the j-space blog post discusses the idea of access consciousness if you are interested. Phenomenal consciousness is what the hard problem of consciousness is about. I personally believe there is a good reason to think the reducibility of this concept to…

Yikes. You're going to have to be more succinct. From my perspective, many of these arguments have the same structure as the arguments against evolution. Long, complex, and unintelligible.

Can we agree your reply is rude given than I have made a fair effort to explain my view to you already? This is not a simple topic and one should not expect the whole thing to come down to a tweet; neither is an accurate understanding of evolution I’ll add.

Which part can I express better? Do you know what I am referring to by phenomenal consciousness?

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#105

Post by qarl »

GPerson wrote: post #104
qarl wrote: post #102

Earlier quoted context omitted.

Yikes. You're going to have to be more succinct. From my perspective, many of these arguments have the same structure as the arguments against evolution. Long, complex, and unintelligible.

Can we agree your reply is rude given than I have made a fair effort to explain my view to you already? This is not a simple topic and one should not expect the whole thing to come down to a tweet; neither is an accurate understanding of evolution I’ll add. Which part can I express better? Do you know what I am referring to by phenomenal consciousness?

I apologize if I've caused offense - I knew it would be contentious but I thought it was more important that you understood my perspective from the outset.

I have looked up "phenomenal consciousness" and I understand it to be roughly the qualia of being conscious. I do not understand then what you meant by "reduce to a substrate independent description".

I hope you can understand why I was so dismissive - your large block of text became unintelligible to me only 5% of the way in.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#106

Post by simonw »

GPerson wrote: post #103
simonw wrote: post #70

Earlier quoted context omitted.

I've been pretty consistent about my disinterest in AGI: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... "You have a very strong track record of speaking out of both sides of your mouth in different settings." If I have a consistent record of that it should be very easy for you to come up with the some examples.

I was considering our interchange yesterday. I apologize for my rude manner in expressing my views. I do appreciate that you have the conscientiousness to engage with your detractors. From my perspective the whole AI Revolution has been deranged in many ways and it has perhaps deranged me as well. I do hold passionate views at odds with some of yours (and many others on this website). It does feel like an existential…

Thanks for saying that.

For what it's worth, I kept engaging because you clearly had a perspective based on real issues and deep consideration.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#107

Post by GPerson »

qarl wrote: post #105
GPerson wrote: post #104

Earlier quoted context omitted.

Can we agree your reply is rude given than I have made a fair effort to explain my view to you already? This is not a simple topic and one should not expect the whole thing to come down to a tweet; neither is an accurate understanding of evolution I’ll add. Which part can I express better? Do you know what I am referring to by phenomenal consciousness?

I apologize if I've caused offense - I knew it would be contentious but I thought it was more important that you understood my perspective from the outset. I have looked up "phenomenal consciousness" and I understand it to be roughly the qualia of being conscious. I do not understand then what you meant by "reduce to a substrate independent description". I hope you can understand why I was so dismissive - your large…

I’m not more offended than by the army of people here informing me that my IQ is low, so no worries. I don’t have the skill to express these ideas more succinctly, and I apologize for this. I’ve tangled a lot, so I can try to express some parts in more organized manner if you’ll bear with me.

(1) An important topic in consciousness studies is Chalmers’ hard problem; the original essay is very readable and I recommend it. He sets it aside from the “easy” problems of explaining how various human functions correspond to neural mechanisms, since he supposes these are clearly achievable by science with time. Many find this name contentious or the framing misguided. The hard problem is to explain how and why phenomenal consciousness exists, i.e., that there is an experience that goes along with some of those neural mechanisms. This is perplexing because if the functional aspects can all be explained mechanically, there is no a priori reason there must be an experience.

(2) Substrate independence is the quality of the essential features of a process to depend only on their information content, not on the underlying medium expressing the information. For instance, the working of an operating system is substrate independent in the sense that it can run on different hardware. It is a commonly held belief that consciousness is like this, and I suspected this was your view. My own belief is that fundamental physical processes are substrate dependent. Each assumption has different explanatory power and issues.

I fear this is not succinct but my urge is to continue.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#108

Post by qarl »

GPerson wrote: post #107
qarl wrote: post #105

Earlier quoted context omitted.

I apologize if I've caused offense - I knew it would be contentious but I thought it was more important that you understood my perspective from the outset. I have looked up "phenomenal consciousness" and I understand it to be roughly the qualia of being conscious. I do not understand then what you meant by "reduce to a substrate independent description". I hope you can understand why I was so dismissive - your large…

I’m not more offended than by the army of people here informing me that my IQ is low, so no worries. I don’t have the skill to express these ideas more succinctly, and I apologize for this. I’ve tangled a lot, so I can try to express some parts in more organized manner if you’ll bear with me. (1) An important topic in consciousness studies is Chalmers’ hard problem ; the original essay is very readable and I recommen…

> I fear this is not succinct but my urge is to continue.

Regardless, I understood all of it, and I thank you for sticking it out with me.

Re (1) - yes, I am familiar. To cut to the chase - I agree that P-Zombies can never be strictly ruled out, but I find this issue to be akin to solipsism.

Re (2) - yes, that does describe me: neurons are no more conscious-making than transistors. But again, I agree this can never be proven strictly.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#109

Post by GPerson »

qarl wrote: post #108
GPerson wrote: post #107

Earlier quoted context omitted.

I’m not more offended than by the army of people here informing me that my IQ is low, so no worries. I don’t have the skill to express these ideas more succinctly, and I apologize for this. I’ve tangled a lot, so I can try to express some parts in more organized manner if you’ll bear with me. (1) An important topic in consciousness studies is Chalmers’ hard problem ; the original essay is very readable and I recommen…

> I fear this is not succinct but my urge is to continue. Regardless, I understood all of it, and I thank you for sticking it out with me. Re (1) - yes, I am familiar. To cut to the chase - I agree that P-Zombies can never be strictly ruled out, but I find this issue to be akin to solipsism. Re (2) - yes, that does describe me: neurons are no more conscious-making than transistors. But again, I agree this can never b…

No, thank you for engaging.

The most naive versions of substrate independence have a hard time contending with Putnam’s thought experiment showing rocks are conscious (Chalmers has a webpage discussing this also); the idea is that a sequence of machine states can be made to correspond 1-1 to a sequence of rock states. There are stronger versions which start making assumptions about causal and organizational structure, which is probably your view. This suggests at least that consciousness depends not only on information expression, but the manner in which the information expression happens. This doesn’t preclude your position, but it starts to sound like the substrate might matter because certain causal and organizational architectures may be only expressible in our reality in certain ways.

I could talk your ear off though.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#110

Post by qarl »

GPerson wrote: post #109
qarl wrote: post #108

Earlier quoted context omitted.

> I fear this is not succinct but my urge is to continue. Regardless, I understood all of it, and I thank you for sticking it out with me. Re (1) - yes, I am familiar. To cut to the chase - I agree that P-Zombies can never be strictly ruled out, but I find this issue to be akin to solipsism. Re (2) - yes, that does describe me: neurons are no more conscious-making than transistors. But again, I agree this can never b…

No, thank you for engaging. The most naive versions of substrate independence have a hard time contending with Putnam’s thought experiment showing rocks are conscious (Chalmers has a webpage discussing this also); the idea is that a sequence of machine states can be made to correspond 1-1 to a sequence of rock states. There are stronger versions which start making assumptions about causal and organizational structure…

This rocks thing sounds like an argument I heard Searle make one time - that the vibrating molecules in a wall can be thought of as sentient because you can map them to intelligent output (where, of course, the intelligence is actually hidden in the mapping.)

I don't know how to resolve these things - but my suspicion is these issues get resolved like the question of solipsism: we take a practical explanation because it's so useful.

Because, at the end of the day, you're going to be hard-pressed to prove consciousness when it comes from a circular definition.

I look to vitalism for inspiration. I am hard pressed to find inexplicable behavior in evolved chemistry. It remains chemistry.

Probably.

Post reply on HN