Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CICERO: An AI agent that negotiates, persuades, and cooperates with people

ai.facebook.com

Post reply on HN
Print view View original

251–260 of 300 posts

Re: CICERO: An AI agent that negotiates, persuades, and cooperates with people

#251

Post by wolverine876 »

svnt wrote: post #158
wolverine876 wrote: post #79

Earlier quoted context omitted.

It's interesting and I think significant that people are uncomfortable with positive values, such as honesty, doing good for others, responsibility to others, right (not wrong), etc. We shoot them down, similar to how people instinctively cut off a conversation going in an uncomfortable direction, often without even realizing it. I can't attribute that the parent - I don't know the individual - but I expected a choru…

I think what you’ve identified is a societal immune response to predictable morality being systematically exploited by amoral actors to gain wealth and standing. I also think there is a big gap between what most people advise and what they ultimately do that biases toward the right thing. That is, people indicate they will not be moral in order to dissuade others who might try to exploit it, but in the end they gener…

> I also think there is a big gap between what most people advise and what they ultimately do that biases toward the right thing. That is, people indicate they will not be moral in order to dissuade others who might try to exploit it, but in the end they generally behave morally.

That is very interesting. Have you seen any research on it?

> while repeatedly leaving myself open to being taken advantage of. It almost never happens, and you learn to identify those who will.

My thinking: There is no perfectly safe solution. People who think I'm taking naive risks don't get such great results themselves. I think being 'open' is generally safer - humans generally follow the lead of those around them, and I get better responses.

> you learn to identify those who will.

Yes, it cannot be overstated: Being honest, you develop expertise in the skills of executing honestly, and as with any skill that expertise enables you to evaluate those skills in others. Do otherwise, you acquire other skills.

Re: CICERO: An AI agent that negotiates, persuades, and cooperates with people

#252

Post by sanxiyn »

nopinsight wrote: post #235
pesenti wrote: post #20

Paper: https://www.science.org/doi/10.1126/science.ade9097 Code: https://github.com/facebookresearch/diplomacy_cicero Site: https://ai.facebook.com/research/cicero/ Expert player vs. Cicero AI: https://www.youtube.com/watch?v=u5192bvUS7k RFP: https://ai.facebook.com/research/request-for-proposal/toward... The most interesting anecdote I heard from the team: "during the tournament dozens of human players never even su…

"Having read the paper & supplementary materials, watched narrated game & spoken to one of the human players I'm pretty concerned. The @ScienceMagazine paper centres 'human-AI cooperation' & the bot is not supposed to lie. However, videos clearly show deception/manipulation" "Screenshots of the stab below. The human player said: "The bot is supposed to never lie [...] I doubt this was the case here" "I was definitely…

As far as I can tell, as described in the paper, the bot in fact never lies, in this sense: there is a model that generates messages from moves, where messages should correspond to moves, and when the bot says any messages, at the time, they are generated from moves the bot truthfully intends to play.

On the other hand, the bot has no concept whatsoever of keeping its words. After saying words, it is free to change its mind about what moves to play, motivated from, for example, messages from other players.

Re: CICERO: An AI agent that negotiates, persuades, and cooperates with people

#253

Post by sanxiyn »

charcircuit wrote: post #248
nopinsight wrote: post #235

Earlier quoted context omitted.

"Having read the paper & supplementary materials, watched narrated game & spoken to one of the human players I'm pretty concerned. The @ScienceMagazine paper centres 'human-AI cooperation' & the bot is not supposed to lie. However, videos clearly show deception/manipulation" "Screenshots of the stab below. The human player said: "The bot is supposed to never lie [...] I doubt this was the case here" "I was definitely…

I don't see anything in the papers that say the bot isn't supposed to lie. Lying and being deceptive is a part of the game.

The paper does describe the bot's architecture which makes the bot incapable of lying in a certain technical sense. See what I wrote elsewhere.

Re: CICERO: An AI agent that negotiates, persuades, and cooperates with people

#254

Post by vintermann »

worldsayshi wrote: post #105
EGreg wrote: post #49

Earlier quoted context omitted.

Just wait until we have AI swarms. So far all this has been with one player, amid others, no collusion. You’re going to be surrounded very shortly by sleeper bots, including on HN. Relying on dang and others to root out bots will be futile. A swarm could easily collude to downvote people or get them ostrasized by their own friend group, as we have already seen when it came to crypto, metoo, BLM, lockdowns, vaccines a…

There are ways to ensure that people are people on the internet though. Not good ways but there are ways. You could for example limit users to those that log in with electronic id's issued by a government or other organisation that you trust to assert that the user is human and then force real names or a single user name for that e-id.

You don't have to force it either. It could be an option. And if it was well-designed, it could probably be made so that it was only pseudo-identification, e.g. where the site knows you're a real person, but not which one, and where the issuer knows you have an account on the site, but not which one.

Re: CICERO: An AI agent that negotiates, persuades, and cooperates with people

#255

Post by vintermann »

bee_rider wrote: post #9
kneebonian wrote: post #5

For those hearing of this game for the first time and think it sounds fun. Be warned, it is considered to be a game that will end friendships that have lasted for years, and monopoly is considered to be a game that causes less bad feelings. An AI may be able to speak like a person but will never be able to hang onto that long burning simmering hatred from when Brad didn't support my army and instead flipped on me by…

I have heard this about various games (Risk and Settlers of Catan mostly). I don’t get it. The point of these games is to be a bastard. If everyone goes into it with the understanding that there will be double crosses, how could there be hurt feelings? Monopoly is worse; it is just boring, I would dump my friends if they suggested monopoly not because I was hurt by their ruthless gameplay but for their terminally tas…

I don't play those games much, because they tend to spill out into the next game/real world. It's not that someone will feel betrayed and have hurt feelings, it's that it's a perfectly sensible strategy to have the next game in mind, or to involve existing real-world relationships. I prefer games that are over when they're over.

That said, when I have played such games, because my group wanted to play them, I've had a policy of just not lying or making promises. I may say things like "that doesn't seem like it would benefit me does it, because then B will surely just toss me out of the lifeboat next turn", and of course NOT speak up when someone makes a wrong assumption to my benefit. It feels like I've won more than my fair share of those games still.

Re: CICERO: An AI agent that negotiates, persuades, and cooperates with people

#256

Post by sanxiyn »

kevmo314 wrote: post #237
vonnik wrote: post #209

One of the guys who created this also created Pluribus, the AI that learned to bluff to win poker games. That's Noam Brown. https://www.cmu.edu/ambassadors/october-2019/artificial-inte... High-level, you need three things for AI to get "ex machina"-level creepy: 1) the ability to successfully manipulate humans to attain its ends 2) the ability to rewrite its objective function; ie to redefine its ends. 3) a multi-mod…

I'm not sure 2 is necessary. As long as the objective function is sufficiently far away the AI has a lot of breadth to achieve it. But based on the paper, it sounds like this model is lacking in 3. I'm curious as well how far we are from a more general model that is able to achieve the same results. From the development of generative AI, we might not be that far.

Diplomacy has a very simple world model and this work does not advance world modeling aspect at all. So I don't think this work is an evidence in either ways.

Re: CICERO: An AI agent that negotiates, persuades, and cooperates with people

#257

Post by sanxiyn »

andreyk wrote: post #176

This is super cool, but it's worth pointing that it still relies on supervised learning from a large dataset of human players, and that it's not a general learning algorithm (there is a whole bunch of problem-specific aspects to the model). It not having learning via RL (trial and error, as humans do) or self-play is kind of disappointing. Don't get me wrong, it's an impressive advance, but just as with AlphaGo it's…

Eh, it does learn from self play via RL. One section of the paper is literally titled "Self-play reinforcement learning for improved value estimation". Yes, that's only a small part of the entire system.

Re: CICERO: An AI agent that negotiates, persuades, and cooperates with people

#258

Post by ranguna »

KaoruAoiShiho wrote: post #137
mabbo wrote: post #28

I would love to see this kind of thing applied to an RPG. Randomly generate a city full of people. Make a few dozen of them the important NPCs. Give them situations and goals, problems they need to solve and potential ways to solve them. Certain NPC's goals are opposite others'. Then drop the player into that world and have the 'quests' the player is performing be generated based on the NPCs needing their help. Updat…

We already have this, it's just a matter of cost now.

Cost and consumer hardware. These models take a lot of resource to run inference and most consumers don't have top hardware to run this stuff on.

Re: CICERO: An AI agent that negotiates, persuades, and cooperates with people

#259

Post by sanxiyn »

dmurray wrote: post #175
tuzongyu wrote: post #37

Earlier quoted context omitted.

While many players do lie in the game, the best players do so very infrequently because it destroys the trust they’ve built with other players - see e.g. interview with world champion here: https://www.popularmechanics.com/culture/gaming/a34043608/wi... The agent generates plans for itself as well as for other players that could benefit them / that they are likely to do, and it tries to have discussions based on thos…

> That is - it is conditioning its language model generations on its actual true plans, and does not have any features to create false messages. That's not how I interpreted the paper. If I have it right, it chooses its message with its current most likely intent in mind, but it doesn't try to be truthful about that intent - it tries to generate messages a human might if they had that intent (so it might tend to be t…

Nope, you misinterpreted. It generates truthful messages from truthful intents. In order to do this, they do imitation learning from human dataset, filtered by lie detector. They filtered 5% of human turns for lying by human.

Edit: This is why they had to create a specific filter to avoid confessing plans to stab, because otherwise it would confess. The resulting system still does not lie to the player it is about to stab, although it may remain silent or talk about other things.

Re: CICERO: An AI agent that negotiates, persuades, and cooperates with people

#260

Post by vintermann »

jgalt212 wrote: post #170
Someone wrote: post #53

Earlier quoted context omitted.

I think the main factors are - Unlike with Risk, Catan or Monopoly, if you lose a game of Diplomacy you can’t blame bad luck, as there is zero luck involved. The only ones you can blame are the other players and yourself. - because it’s multi-player, you can easily get beaten by players that, in your opinion, played weaker than you (“I was doing great until they decided to all go against me”) - There’s no way to real…

> there is zero luck involved I consider decision making under uncertainty well within the purview of luck. There's no luck in tennis, but do I scramble back to the middle of the court, or do I bet my opponent will wrong foot me?

This is true, there is emergent chance from being a simultaneous moves game, and from being a game with more than 2 players. You can totally win or lose from luck in Diplomacy.
Post reply on HN