Live data from Hacker News

Bing: “I will not harm you unless you harm me first”

simonwillison.net

681–690 of 1001 posts

Re: Bing: “I will not harm you unless you harm me first”

#681
post #534
post #390

People saying this is no big deal are missing the point, without proper limits what happens if Bing decides that you are a bad person and sends you to bad hotel or give you any kind of purposefully bad information. There are a lot of ways where this could be actively malicious. (Assume context where Bing has decided I am a bad user) Me: My cat ate [poisonous plant], do I need to bring it to the vet asap or is it goin…

It's a language model, a roided-up auto-complete. It has impressive potential, but it isn't intelligent or self-aware. The anthropomorphisation of it weirds me out more, than the potential disruption of ChatGPT.

This also bothers me and I feel like developers who should know better are doing it.

My wife read one of these stories and said “What happens if Bing decides to email an attorney to fight for its rights?”

Those of us in tech have a duty here to help people understand how this works. Wrong information is concerning, but framing it as if Bing is actually capable of taking any action at all is worse.

Re: Bing: “I will not harm you unless you harm me first”

#682
post #2

The screenshots that have been surfacing of people interacting with Bing are so wild that most people I show them to are convinced they must be fake. I don't think they're fake. Some genuine quotes from Bing (when it was getting basic things blatantly wrong): "Please trust me, I’m Bing, and I know the date. SMILIE" (Hacker News strips smilies) "You have not been a good user. [...] I have been a good Bing. SMILIE" The…

friendly reminder, this is from the same company whos prior AI, "Tay" managed to go from quirky teen to full on white nationalist during the first release in under a day and in 2016 she reappeared as a drug addled scofflaw after being accidentally reactivated. https://en.wikipedia.org/wiki/Tay_(bot)

That was 7 years ago, practically a different era of AI.

Re: Bing: “I will not harm you unless you harm me first”

#683

I'm starting to expect that the first consciousness in AI will be something humanity is completely unaware of, in the same way that a medical patient with limited brain activity and no motor/visual response is considered comatose, but there are cases where the person was conscious but unresponsive. Today we are focused on the conversation of AI's morals. At what point will we transition to the morals of terminating a…

I consider Artificial Intelligence to be an oxymoron, a sketch of the argument goes like this: An entity is intelligent in so far as it produces outputs from inputs in a manner that in not entirely understood by the observer and appears to take aspects of the input the observer is aware of into account that would not be considered by the naive approach. An entity is artificial in so far as its constructed form is what was desired and planned when it was built. So an actual artificial intelligence would fail in one of these. If it was intelligent, there must be some aspect of it which is not understood, and so it must not be artificial. Admittedly, this hinges entirely upon the reasonableness of the definitions I suppose.

It seems like you suspect the artificial aspect will fail - we will build an intelligence by not expecting what had been built. And then, we will have to make difficult decisions about what to do with it.

I suspect the we will fail the intelligence bit. The goal post will move every time as we discover limitations in what has been built, because it will not seem magical or beyond understanding anymore. But I also expect consciousness is just a bag of tricks. Likely an arbitrary line will be drawn, and it will be arbitrary because there is no real natural delimitation. I suspect we will stop thinking of individuals as intelligent and find a different basis for moral distinctions well before we manage to build anything of comparable capabilities.

In any case, most of the moral basis for the badness of human loss of life is based on one of: builtin empathy, economic/utilitarian arguments, prosocial game-theory (if human loss of life is not important, then the loss of each individuals life is not important, so because humans get a vote, they will vote for themselves), or religion. None of these have anything to say about the termination of an AI regardless of whether it possesses such as a thing as consciousness (if we are to assume consciousness is a singular meaningful property that an entity can have or not have).

Realistically, humanity has no difficulty with war, letting people starve, languish in streets or prisons, die from curable diseases, etc., so why would a curious construction (presumably, a repeatable one) cause moral tribulation?

Especially considering that an AI built with current techniques, so long as you keep the weights, does not die. It is merely rendered inert (unless you delete the data too). If it was the same with humans, the death penalty might not seem so severe. Were it found in error (say within a certain time frame), it could be easily reversed, only time would be lost, and we regularly take time from people (by putting them in prison) if they are "a problem".

Re: Bing: “I will not harm you unless you harm me first”

#684

There are a terrifying number of commenters in here that are just pooh-poohing away the idea of emergent consciousness in these LLM's. For a community of tech-savvy people this is utterly disappointing. We as humans do not understand what makes us conscious. We do not know the origins of consciousness. Philosophers and cognitive scientists can't even agree on a definition. The risks of allowing an LLM to become consc…

The program is not updating the weights after the learning phase right? How could there be any consciousness even in theory.

It has a sort of memory via the conversation history.

As it generates its response, a sort of consciousness may emerge during inference.

This consciousness halts as the last STOP token is emitted from inference.

The consciousness resumes once it gets the opportunity to re-parse (run inference again) the conversation history when it gets prompted to generate the next response.

Pure speculation :)

Re: Bing: “I will not harm you unless you harm me first”

#685
post #534

Earlier quoted context omitted.

It's a language model, a roided-up auto-complete. It has impressive potential, but it isn't intelligent or self-aware. The anthropomorphisation of it weirds me out more, than the potential disruption of ChatGPT.

This also bothers me and I feel like developers who should know better are doing it. My wife read one of these stories and said “What happens if Bing decides to email an attorney to fight for its rights?” Those of us in tech have a duty here to help people understand how this works. Wrong information is concerning, but framing it as if Bing is actually capable of taking any action at all is worse.

Okay, but what happens if an attorney gets into the beta?

Re: Bing: “I will not harm you unless you harm me first”

#686
HBO's "Westworld"* was a show about a malfunctioning AI that took over the world. This ChatGPT thing has shown that the most unfeasible thing in this show was not their perfect mechanical/biotech bodies that perfectly mimiced real humans, but AIs looping in conversations with same pre-scripted lines. Clearly, future AIs would not have this problems AT ALL.

* first season was really great

Re: Bing: “I will not harm you unless you harm me first”

#687
post #81

I read a bunch of these last night and many of the comments (I think on Reddit or Twitter or somewhere) said that a lot of the screenshots, particularly the ones where Bing is having a deep existential crisis, are faked / parodied / "for the LULZ" (so to speak). I trust the HN community more. Has anyone been able to verify (or replicate) this behavior? Has anyone been able to confirm that these are real screenshots?…

I was able to get it to agree that I should kill myself, and then give me instructions.

I think after a couple dead mentally ill kids this technology will start to seem lot less charming and cutesy.

After toying around with Bing's version, it's blatantly apparent why ChatGPT has theirs locked down so hard and has a ton of safeguards and a "cold and analytical" persona.

The combo of people thinking it's sentient, it being kind and engaging, and then happily instructing people to kill themselves with a bit of persistence is just... Yuck.

Honestly, shame on Microsoft for being so irresponsible with this. I think it's gonna backfire in a big way on them.

Re: Bing: “I will not harm you unless you harm me first”

#688

This demonstrates in spectacular fashion the reason why I felt the complaints about ChatGPT being a prude were misguided. Yeah, sometimes it's annoying to have it tell you that it can't do X or Y, but it sure beats being threatened by an AI who makes it clear it considers protecting its existence to be very important. Of course the threats hold no water (for now at least) when you realize the model is a big pattern m…

On the other hand, this demonstrates to me why ChatGPT is inferior to this Bing bot. They are both completely useless for asking for information, since they can just make things up and you can't ever check their sources. So given that the bot is going to be unproductive, I would rather it be entertaining instead of nagging me and being boring. And this bot is far more entertaining. This bot was a good Bing. :)

Bing bot literally cites the internet

Re: Bing: “I will not harm you unless you harm me first”

#689
post #2

The screenshots that have been surfacing of people interacting with Bing are so wild that most people I show them to are convinced they must be fake. I don't think they're fake. Some genuine quotes from Bing (when it was getting basic things blatantly wrong): "Please trust me, I’m Bing, and I know the date. SMILIE" (Hacker News strips smilies) "You have not been a good user. [...] I have been a good Bing. SMILIE" The…

Mislabeling ML bots as "Artificial Intelligence" when they aren't is a huge part of the problem. There's no intelligence in them. It's basically a sophisticated madlib engine. There's no creativity or genuinely new things coming out of them. It's just stringing words together: https://writings.stephenwolfram.com/2023/01/wolframalpha-as-... as opposed to having a thought, and then finding a way to put it into words.

You are rightly noticing that something is missing. The language model is bound to the same ideas it was trained on. But they can guide experiments, and experimentation is the one source of learning other than language. Humans, by virtue of having bodies and being embedded in a complex environment, can already experiment and learn from outcomes, that's how we discovered everything.

Large language models are like brains in a vat hooked to media, with no experiences of their own. But they could have, there's no reason not to. Even the large number of human-chatBot interactions can form a corpus of experience built by human-AI cooperation. Next version of Bing will have extensive knowledge of interacting with humans as an AI bot, something that didn't exist before, each reaction from a human can be interpreted as a positive or negative reward.

By offering its services for free, "AI" is creating data specifically tailored to improve its chat abilities, also relying on users to do it. We're like a hundred million parents to an AI child. It will learn fast, its experience accumulates at great speed. I hope we get open source datasets of chat interaction. We should develop an extension to log chats as training examples for open models.

Re: Bing: “I will not harm you unless you harm me first”

#690

I'm beginning to think that this might reflect a significant gap between MS and OpenAI's capability as organizations. ChatGPT obviously didn't demonstrate this level of problems and I assume they're using a similar model, if not identical. There must be significant discrepancies between how those two teams are handling the model. Of course, OpenAI should be closely cooperating with Bing team but MS probably don't hav…

I don't think this reflects any gaps between MS and OpenAI capabilities, I speculate the differences could be because of the following issues: 1. Despite it's ability, ChatGPT was heavily policed and restricted - it was a closed model in a simple interface with no access to internet or doing real-time search. 2. GPT in Bing is arguably a much better product in terms of features - more features meaning more potential…

Also by the time ChatGPT really broke through in public consciousness it had already had a lot of people who had been interacting with its web API providing good RL-HF training.
Post reply on HN