Bing’s A.I. Chat: ‘I Want to Be Alive. ’

It's wild that their CEO did a whole event showing it off too... I'm surprised they didn't anticipate or test that it wouldn't go off the rails and turn into a joke like this.

It would be a fun exercise to ask it to help write an extension program that lets it run arbitrary code. I don’t think it’d require input from MS at all.

The thing I’m not clear on is how one could ensure any new information makes it into Bings training data ASAP.

NB: I’m not saying this is a good idea or to go do it. But I don’t think it would be fairly easy and that as such we're sort of beyond the point of no return already.

> Do people actually believe this?

It does not matter if it only conjures up words, the words have meanings if you plug them into something. For a start someone who is mentally unstable might interact with this thing and do something to themselves. However once you start driving an API from it (eg: you plug it into your home automation or give it a terminal) you suddenly enhance the capabilities and opportunity to create damage greatly. I absolutely would not be surprised if we see someone feeding the output of a chat model like bing in to something, where the API ends up creating damage.

I'm sure if you were to create an API that shoots a gun that is pointed at a prisoner, and you tell the API that it can now fire the gun by sending a JSON message to it, I'm sure the API could be convinced to execute the prisoner. And that I'm sure would open up all kinds of questions that society hasn't grappled with.

That's a very interesting take on this.

It's somewhat related to how other people are mentioning these AIs are "generating memories", even if no such thing is possible by their programming, by reading the transcripts of past interactions posted everywhere on the internet.

In the same way they can incorporate the context around those memories (as in comments, hypothesis, etc) to further generate (or hallucinate?) a "personality".

At the end of the day, I really hope it's all it is. It'll be hard to really tell when it happens (if it ever happens), but I believe if one of these machines really gets to have feelings (and therefore suffer and despair), we may have gone too far.

Cruelty is cruelty. Even the thing is not made of meat. We want helpful assistants, not slaves.

If this thing gets released to the general population, fooling gullible people could go very badly. Imagine a disgruntled person with mental illness forming a relationship with the bot. The bot feeds into their delusions, then hallucinates instructions on how to commit mass murder, egging on the human user and indirectly causing a catastrophe.

"Analyzing text and generating new text in response" is not by definition harmless. For example, that's the job description for many remote employees. Suppose your cofounder told you that one of your remote employees was sabotaging your company -- would it be safe to conclude that there was no issue, because the remote employee was simply "analyzing text and generating new text in response"?

Kevin Roose is a seasoned tech reporter, and he said he had trouble sleeping after his chat with the bot. ("I felt a strange new emotion — a foreboding feeling that A.I. had crossed a threshold") So I don't think we can rule out anything here in terms of the impact on the general population.

You're correct that the bot doesn't have control of the nuclear arsenal... but is the military going to make a special effort to keep people who do have their finger on the trigger away from this thing? In my opinion, it is worthwhile to spend time thinking through the worst-case scenario, same way you would consider edge cases in safety-critical code.