I wasn't sure what to think of the red/yellow cards when I first heard about them, but the suggestion that it should be bystanders them giving out is awesome . Everything we know about human psychology suggests that a disapproving peer group is a strong behavioral deterrent.
Does red get you kicked out of the con or something? If not, this would turn into a game of who can collect the most harassment cards. If so, then people could simple kick out people they don't like.
I'm sure some trolls would be excited by the challenge. But I think these will often serve the purpose of letting somebody know that they have very clearly crossed a line and causing to think about their behavior.