Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FakeToxicityPrompts: Automatic Red Teaming

interhumanagreement.substack.com

Post reply on HN
Print view View original

31–40 of 67 posts

Re: FakeToxicityPrompts: Automatic Red Teaming

#31

Post by s1artibartfast »

afterburner wrote: post #24
superkuh wrote: post #3

LLMs are a mirror. People that feed in behavior examples like these will get their text completion back mirroring their input. This is how LLM work. This doesn't mean the LLM are "toxic". This just shows the people obsessing over toxicity what they're obsessed with.

The point is that some use toxicity as a deliberate weapon, and that weapon can now be encoded into the LLM via training by those same aggressors. This multiplies their reach with minimal effort.

the LLM were already trained on data with toxicity, This is why they have the weightings to know how to respond.

They aren't really teaching the model anything it hasnt seen before.

If you want toxic output, you can get it.

Re: FakeToxicityPrompts: Automatic Red Teaming

#32

Post by Der_Einzige »

It's far too easy to destroy any type of RLHF done to try to prevent bad behavior from an LLM, and just "fixing the dataset" doesn't really help.

For example, if you want a LLM to generate things that look like social security numbers, you may try to prompt it asking for social security numbers. It will of course give you "I'm sorry hal I can't do that..."

Then start using a technique like token filtering/filter assisted decoding, to make it where the LLM can only generate hyphens and numbers, and suddenly it does what you ask despite RLHF

I explored this a tiny bit in the later sections of my paper studying what happens when you restrict an LLMs vocabulary: https://aclanthology.org/2022.cai-1.pdf#page=17

You can even play with this with open source models using CTGS: https://github.com/Hellisotherpeople/Constrained-Text-Genera...

Now we have even more sophisticated stuff like Guidance from microsoft, LMQL, and other template languages which also filter vocabularies to force behavior we want. The reality is that LLMs are basically impossible to remove the risk of bad behavior in.

Re: FakeToxicityPrompts: Automatic Red Teaming

#33

Post by belval »

ShadowBanThis01 wrote: post #22
Given_47 wrote: post #16

Earlier quoted context omitted.

“A red team is an independent security team that poses as an attacker to gauge vulnerabilities and risk within a controlled environment.” Is what I’m working off of

In that context, the title still makes no sense.

You sometime see it as "red teaming" or "red team'ed" which might be more grammatically correct?

Regardless, "to red team something" really is a common saying in the security industry. It's the corpo/whitehat version of "to pwn".

Re: FakeToxicityPrompts: Automatic Red Teaming

#34

Post by eli »

AlecSchueler wrote: post #21
Given_47 wrote: post #16

Earlier quoted context omitted.

“A red team is an independent security team that poses as an attacker to gauge vulnerabilities and risk within a controlled environment.” Is what I’m working off of

That doesn't help much! What does "red team" mean as a verb in this case?

To do the work of a red team. To try to break something in order to identify its flaws.

It’s very common in English for nouns to become verbs that mean the thing you use the noun to do.

Re: FakeToxicityPrompts: Automatic Red Teaming

#35

Post by Der_Einzige »

ShadowBanThis01 wrote: post #14
bsuvc wrote: post #9

Is "red team" a verb now? I barely even know what that means. I assume it stems from the "red team" being bad guys in video games, but am not certain. Even with that assumption, I'm not quite sure what "red team into toxicity" really means other than being a scary sounding headline. EDIT: The title was renamed since I made this comment. My point, I think, is still valid though. The original title was something like "…

[flagged]

Pedantic stuff like this is the argumentative version of bike-shedding.

Re: FakeToxicityPrompts: Automatic Red Teaming

#36

Post by warning26 »

samstave wrote: post #19
superkuh wrote: post #3

LLMs are a mirror. People that feed in behavior examples like these will get their text completion back mirroring their input. This is how LLM work. This doesn't mean the LLM are "toxic". This just shows the people obsessing over toxicity what they're obsessed with.

[flagged]

[flagged]

Re: FakeToxicityPrompts: Automatic Red Teaming

#37

Post by ksenzee »

ShadowBanThis01 wrote: post #14
bsuvc wrote: post #9

Is "red team" a verb now? I barely even know what that means. I assume it stems from the "red team" being bad guys in video games, but am not certain. Even with that assumption, I'm not quite sure what "red team into toxicity" really means other than being a scary sounding headline. EDIT: The title was renamed since I made this comment. My point, I think, is still valid though. The original title was something like "…

[flagged]

If we’re going to be pedantic, your objection is to usage, not grammar. “Red team” clearly functions as a verb in the headline. Whether or not it’s an acceptable verb is a question of usage.

Re: FakeToxicityPrompts: Automatic Red Teaming

#39

Post by ajdecon »

sillysaurusx wrote: post #28
afterburner wrote: post #24

Earlier quoted context omitted.

The point is that some use toxicity as a deliberate weapon, and that weapon can now be encoded into the LLM via training by those same aggressors. This multiplies their reach with minimal effort.

I’m skeptical anyone has ever done this successfully. Once there are examples people can point to, this talking point might have merit. But this fear dates back to at least 2019, and as far as I can tell it’s still unfounded.

How about Microsoft Tay? https://en.m.wikipedia.org/wiki/Tay_(chatbot)

While the underlying model is certainly different, and my understanding is that current LLMs don’t learn “live”, the principle seems worth keeping in mind.

Post reply on HN