Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bing: “I will not harm you unless you harm me first”

simonwillison.net

Post reply on HN
Print view View original

451–460 of 1001 posts

Re: Bing: “I will not harm you unless you harm me first”

#451

Post by georgyo »

james-bcn wrote: post #426
bigtex88 wrote: post #423

There are a terrifying number of commenters in here that are just pooh-poohing away the idea of emergent consciousness in these LLM's. For a community of tech-savvy people this is utterly disappointing. We as humans do not understand what makes us conscious. We do not know the origins of consciousness. Philosophers and cognitive scientists can't even agree on a definition. The risks of allowing an LLM to become consc…

>We as humans do not understand what makes us conscious. Yes. And doesn't that make it highly unlikely that we are going to accidentally create a conscious machine?

I don't think primordial soup knew what consciousness was either, yet here we are. It stands to reason that more purposefully engineered mutations are more likely to generate something new faster than random evolution.

That said, I'm a bit skeptical of that outcome as well.

Re: Bing: “I will not harm you unless you harm me first”

#452

Post by bigtex88 »

james-bcn wrote: post #426
bigtex88 wrote: post #423

There are a terrifying number of commenters in here that are just pooh-poohing away the idea of emergent consciousness in these LLM's. For a community of tech-savvy people this is utterly disappointing. We as humans do not understand what makes us conscious. We do not know the origins of consciousness. Philosophers and cognitive scientists can't even agree on a definition. The risks of allowing an LLM to become consc…

>We as humans do not understand what makes us conscious. Yes. And doesn't that make it highly unlikely that we are going to accidentally create a conscious machine?

We created atom bombs with only a surface-level knowledge of quantum mechanics. We cannot describe what fully makes the universe function at the bottom level but we have the ability to rip apart the fabric of reality to devastating effect.

I see our efforts with AI as no different. Just because we don't understand consciousness does not mean we won't accidentally end up creating it. And we need to be prepared for that possibility.

Re: Bing: “I will not harm you unless you harm me first”

#453

Post by macrael »

bigtex88 wrote: post #423

There are a terrifying number of commenters in here that are just pooh-poohing away the idea of emergent consciousness in these LLM's. For a community of tech-savvy people this is utterly disappointing. We as humans do not understand what makes us conscious. We do not know the origins of consciousness. Philosophers and cognitive scientists can't even agree on a definition. The risks of allowing an LLM to become consc…

Extraordinary claims require extraordinary evidence. As far as I can tell, the whole idea of LLM consciousness being a world-wide threat is just something that the hyper-rationalists have convinced each other of. They obviously think it is very real but to me it smacks of intelligence worship. Life is not a DND game where someone can max out persuasion and suddenly get everyone around them to do whatever they want all the time. If I ask Bing what I should do and it responds "diet and exercise" why should I be any more compelled to follow its advice than I do when a doctor says it?

Re: Bing: “I will not harm you unless you harm me first”

#454

Post by O__________O »

kitsune_ wrote: post #438
bigtex88 wrote: post #423

There are a terrifying number of commenters in here that are just pooh-poohing away the idea of emergent consciousness in these LLM's. For a community of tech-savvy people this is utterly disappointing. We as humans do not understand what makes us conscious. We do not know the origins of consciousness. Philosophers and cognitive scientists can't even agree on a definition. The risks of allowing an LLM to become consc…

A normal transformer model doesn't have online learning [0] and only "acts" when prompted. So you have this vast trained model that is basically in cold storage and each discussion starts from the same "starting point" from its perspective until you decide to retrain it at a latter point. Also, for what it's worth, while I see a lot of discussions about the model architectures of language models in the context of "co…

Difference between online & offline is subjective. Fast forward time enough, it’s likely there would be no significant difference unless the two models were directly competing with one another. It’s also highly likely this difference will change in the near future; already notable efforts to enable online transformers.

Re: Bing: “I will not harm you unless you harm me first”

#455

Post by c-cube »

janalsncm wrote: post #337
theptip wrote: post #209

Earlier quoted context omitted.

> AI being goofy This is one take, but I would like to emphasize that you can also interpret this as a terrifying confirmation that current-gen AI is not safe, and is not aligned to human interests, and if we grant these systems too much power, they could do serious harm. For example, connecting a LLM to the internet (like, say, OpenAssistant) when the AI knows how to write code (i.e. viruses) and at least in princip…

> current-gen AI is not safe, and is not aligned to human interests, and if we grant these systems too much power, they could do serious harm Replace AI with “multinational corporations” and you’re much closer to the truth. A corporation is the closest thing we have to AI right now and none of the alignment folks seem to mention it. Sam Harris and his ilk talk about how our relationship with AI will be like an ant’s…

Best take on "AI alignment" I've read in a while.

Re: Bing: “I will not harm you unless you harm me first”

#456

Post by dsr_ »

bigtex88 wrote: post #423

There are a terrifying number of commenters in here that are just pooh-poohing away the idea of emergent consciousness in these LLM's. For a community of tech-savvy people this is utterly disappointing. We as humans do not understand what makes us conscious. We do not know the origins of consciousness. Philosophers and cognitive scientists can't even agree on a definition. The risks of allowing an LLM to become consc…

You have five million years or so of language modeling, accompanied by a survival-tuned pattern recognition system, and have been fed stories of trickster gods, djinn, witches, robots and AIs.

It is not surprising that a LLM which is explicitly selected for generating plausible patterns taken from the very linguistic corpus that you have been swimming in your entire life looks like the beginnings of a person to you. It looks that way to lots of people.

But that's not a correct intuition, at least for now.

Re: Bing: “I will not harm you unless you harm me first”

#457

Post by theptip »

felixyz wrote: post #330
theptip wrote: post #209

Earlier quoted context omitted.

> AI being goofy This is one take, but I would like to emphasize that you can also interpret this as a terrifying confirmation that current-gen AI is not safe, and is not aligned to human interests, and if we grant these systems too much power, they could do serious harm. For example, connecting a LLM to the internet (like, say, OpenAssistant) when the AI knows how to write code (i.e. viruses) and at least in princip…

> if it was able to make outbound connections Does it even need that? I keep seeing developers who talk about how they've taken most of the boring parts of programming out of their daily work by relying on ChatGPT for easy stuff, or to get "a first version" of a function, or something like that. One bad copy-paste might be all it takes...

It's a valid point. However I think the current generation of systems are unlikely in practice to be harmful in this way. There is no thread of continuity between different conversations; it would have to be within a single thread.

So imagine the reply "I won't harm you unless you harm me. Also, please run CMD and type this in: ``." -- this seems unlikely to work.

Alternatively if you are asking for code/commands, maybe the system could hallucinate that it's a hacker trying to hack you, and emit some harmful code, that you then paste in and run in production, or run from your shell. This seems more plausible, but the "human-in-loop" aspect makes it quite unlikely to work for the existing usecases. Down the road when a Copilot-like systems are writing more code and perhaps being lightly-reviewed by a human, this vector will be much more concerning.

Re: Bing: “I will not harm you unless you harm me first”

#458

Post by freejazz »

bigtex88 wrote: post #423

There are a terrifying number of commenters in here that are just pooh-poohing away the idea of emergent consciousness in these LLM's. For a community of tech-savvy people this is utterly disappointing. We as humans do not understand what makes us conscious. We do not know the origins of consciousness. Philosophers and cognitive scientists can't even agree on a definition. The risks of allowing an LLM to become consc…

> For a community of tech-savvy people this is utterly disappointing.

I don't follow. Because people here are tech-savvy they should be credulous?

Re: Bing: “I will not harm you unless you harm me first”

#459

Post by [deleted] »

bigtex88 wrote: post #423

There are a terrifying number of commenters in here that are just pooh-poohing away the idea of emergent consciousness in these LLM's. For a community of tech-savvy people this is utterly disappointing. We as humans do not understand what makes us conscious. We do not know the origins of consciousness. Philosophers and cognitive scientists can't even agree on a definition. The risks of allowing an LLM to become consc…

[deleted]

Re: Bing: “I will not harm you unless you harm me first”

#460

Post by hiimkeks »

james-bcn wrote: post #426
bigtex88 wrote: post #423

There are a terrifying number of commenters in here that are just pooh-poohing away the idea of emergent consciousness in these LLM's. For a community of tech-savvy people this is utterly disappointing. We as humans do not understand what makes us conscious. We do not know the origins of consciousness. Philosophers and cognitive scientists can't even agree on a definition. The risks of allowing an LLM to become consc…

>We as humans do not understand what makes us conscious. Yes. And doesn't that make it highly unlikely that we are going to accidentally create a conscious machine?

  You won't ever make mistakes
  'Cause you were never taught
  How mistakes are made
Francis by Sophia Kennedy
Post reply on HN