The naughty username checking system used by Twitch • Hacker News Board

Re: The naughty username checking system used by Twitch

#101

Post by b112 » Thu, Oct 07, 2021, 5:11 AM UTC

rhema wrote: post #5

I have a hard time believing this was / is the real version used. It doesn't seem broad enough. More likely it was a kind of smoketest that made sure that a more automated keyword checker was working. It does remind me of the XKEYSCORE (Snowden leaks) that used keywords to bubble up potential threats from emails etc https://www.businessinsider.com/nsa-prism-keywords-for-domes... .

If valid, this means virtually every phone call, and every email (with clients) is flagged. :P

This must be more nuanced. Maybe, it's "additional algo processing when a word is hit", eg another layer before "involve human".

Re: The naughty username checking system used by Twitch

#102

Post by deepstack » Thu, Oct 07, 2021, 5:18 AM UTC

would have being better to have the data source in a json/csv formate, like an array of RegExpression. SQL is not compatible with different dbs.

Re: The naughty username checking system used by Twitch

#103

Post by djrockstar1 » Thu, Oct 07, 2021, 5:20 AM UTC

ancharm wrote: post #36

Is there a blog or something where someone is going through the dump and summarizing?

Your best bet atm is to just look through reddit/hn comments/posts people make as they find stuff. The leak's too big for one person/team to quickly find all spicy stuff.

Re: The naughty username checking system used by Twitch

#104

Post by wiradikusuma » Thu, Oct 07, 2021, 5:20 AM UTC

There should be an open source or even Official RFC(tm) for this. The use case is very generic.

Re: The naughty username checking system used by Twitch

#105

Post by ibudiallo » Thu, Oct 07, 2021, 5:24 AM UTC

I used to work for a company that automated customer service. While we were trialing with new a client, our ai service responded to a customer: "Hey B**, thanks for reaching out."

To our surprise, the end user did not feel offended at all. In fact, they were happy because we responded instantly instead of the usual 24 to 48 hours.

[Story]: https://idiallo.com/blog/do-you-make-your-customers-wait

Re: The naughty username checking system used by Twitch

#106

Post by mgdlbp » Thu, Oct 07, 2021, 5:28 AM UTC

More high-profile profanity filters open for viewing: edit filters used in Wikimedia projects, which also together encompass many languages.

The main 'bad words' filter of English Wikipedia:

https://en.wikipedia.org/wiki/Special:AbuseFilter/384

The page (on en:wiki) listing all filters, which also have uses other than detecting abuse:

https://en.wikipedia.org/wiki/Special:AbuseFilter

The special page has the same title in other editions of Wikipedia and other Wikimedia wikis, though many filters are set to hidden. Dutch Wikipedia, for example:

https://nl.wikipedia.org/wiki/Special:AbuseFilter/10

Re: The naughty username checking system used by Twitch

#107

Post by resonious » Thu, Oct 07, 2021, 5:30 AM UTC

beachy wrote: post #93

phamilton wrote: post #34

We had to do this for a link shortening system (to make sure random base64 didn't contain profanity). It was a pretty fun problem. Not just the implementation, but doing the math to make sure it didn't make our shortened links easily enumerable. The implementation wasn't too bad, but we set up logging initially to spit out any random strings it decided to block. I demo'd this in front of the whole company and live ta…

> to make sure random base64 didn't contain profanity I would have said "why bother" until this happened to us. A customer rang us up in a fury because some demo/ random data that we generated happened to have the word "penis" in it. They were convinced we must have put it there because we thought he was a cock. It was very difficult to defuse the situation.

I just recently saw a randomly generated ID of ours in production that starts with "doggy". Thankfully "doggy" is pretty innocent, but it really made me think "wow what if it was something bad". Unfortunate that that exact scenario seems to have happened to you already.

Re: The naughty username checking system used by Twitch

#108

Post by staysafeanon » Thu, Oct 07, 2021, 5:30 AM UTC

Actually in Twitch's codebase:

  OR replace($1,'_','') SIMILAR TO '%(hate|kill|keel|hang|burn|gasthe)%(black|bl4ck|black|jew|trans|gay|african|afrikan|minorit|asian|nig|n1g)%'

Twitch: "Hatred against Blacks, Jews, Asians, trans? BANNED! Hatred against Whites? I'LL ALLOW IT!"

Re: The naughty username checking system used by Twitch

#109

Post by ademarre » Thu, Oct 07, 2021, 5:33 AM UTC

xdfgh1112 wrote: post #96

ademarre wrote: post #72

Earlier quoted context omitted.

When it comes to censoring randomly generated strings, I like simply to omit vowels from the alphabet. Usually I'll omit some of the more obvious lookalikes too, e.g. [1 0 v]. It's a simple solution. Sure, it is still possible for something to slip through that looks similar to something bad. But the potential to strongly offend is greatly reduced.

Gov.uk did this. The code WNKR still ended up on Reddit yesterday.

That's amusing, but I think it also highlights the effectiveness of the strategy. WNKR is excusable and defensible. WANK would not be.

Edit: But I'll concede that when your outputs are only four characters long and end users will actively interact with them (write them down, type them again later, etc.), additional safeguards might be appropriate. Or simply omit all alphas and use only numerics.

Re: The naughty username checking system used by Twitch

#110

Post by for1nner » Thu, Oct 07, 2021, 5:34 AM UTC

Amusing to see the is_hateful query which used to just be an unreadable mess of regex for validation. Clearly the infra has grown/changed over 10 years, and this is obviously neater, but boy does it LOOK a lot bigger now (and I'm sure they've added yet more new words that try to hurt people...)