The naughty username checking system used by Twitch • Hacker News Board

Re: The naughty username checking system used by Twitch

#61

Post by cbsmith » Thu, Oct 07, 2021, 3:06 AM UTC

DethNinja wrote: post #56

Why not just convert numbers like 1 to i or l then check with a manually created bad word list? Would regex be really much faster than checking it against a 1000 or more bad word list? Also bad word list can easily get updated by moderators as well, I really can’t understand the logic behind using so much regex.

A bad words list is a regex. ;-)

But note: for the most part this isn't using regexs, and to the extent it does, it seems largely intended to make the maintainers' lives easier by avoiding having to represent (and maintain) all the permutations they are trying to match for.

What's sad though is that they're doing many, many passes through the pattern matcher, rather than just building a single big DFA from the whole list of patterns they want to match, which gets traversed in one pass.

Re: The naughty username checking system used by Twitch

#62

Post by tclancy » Thu, Oct 07, 2021, 3:06 AM UTC

sorenjan wrote: post #55

Reminds me of the guy that streamed a talking banana on Twitch, where viewers could make it say things. People submitted variations of the n-word and got him banned, and after trying to filter out all character combinations he could think of he wrote a phonetic filter. That apparently worked much better than trying to think of every permutation of characters that sounds like bad words. https://youtu.be/bJ5ppf0po3k?t=…

They figured that out on Ellis Island so yeah. Soundex.

Re: The naughty username checking system used by Twitch

#63

Post by EvanAnderson » Thu, Oct 07, 2021, 3:07 AM UTC

pchristensen wrote: post #47

phamilton wrote: post #34

We had to do this for a link shortening system (to make sure random base64 didn't contain profanity). It was a pretty fun problem. Not just the implementation, but doing the math to make sure it didn't make our shortened links easily enumerable. The implementation wasn't too bad, but we set up logging initially to spit out any random strings it decided to block. I demo'd this in front of the whole company and live ta…

We did a similar thing at Groupon after a customer’s coupon code contained an F bomb.

I removed the letter U from a random password generator for a Customer's app after a password was generated containing the "C-word".

Re: The naughty username checking system used by Twitch

#64

Post by geoduck14 » Thu, Oct 07, 2021, 3:10 AM UTC

sorenjan wrote: post #55

Reminds me of the guy that streamed a talking banana on Twitch, where viewers could make it say things. People submitted variations of the n-word and got him banned, and after trying to filter out all character combinations he could think of he wrote a phonetic filter. That apparently worked much better than trying to think of every permutation of characters that sounds like bad words. https://youtu.be/bJ5ppf0po3k?t=…

>the guy that streamed a talking banana on Twitch

Of all the ways I expected a talking banana to backfire, I didn't expect this one. Thanks for sharing

Re: The naughty username checking system used by Twitch

#65

Post by geoduck14 » Thu, Oct 07, 2021, 3:12 AM UTC

phamilton wrote: post #34

We had to do this for a link shortening system (to make sure random base64 didn't contain profanity). It was a pretty fun problem. Not just the implementation, but doing the math to make sure it didn't make our shortened links easily enumerable. The implementation wasn't too bad, but we set up logging initially to spit out any random strings it decided to block. I demo'd this in front of the whole company and live ta…

>the first one that popped up during the demo was a big ole F bomb

And this, ladies and gentlemen, is what it would show BEFORE the filter... but after (runs the code again, and prays it works) ... NO PROFANITY!

Re: The naughty username checking system used by Twitch

#66

Post by AmericanChopper » Thu, Oct 07, 2021, 3:13 AM UTC

mc32 wrote: post #11

rhema wrote: post #5

I have a hard time believing this was / is the real version used. It doesn't seem broad enough. More likely it was a kind of smoketest that made sure that a more automated keyword checker was working. It does remind me of the XKEYSCORE (Snowden leaks) that used keywords to bubble up potential threats from emails etc https://www.businessinsider.com/nsa-prism-keywords-for-domes... .

It also mostly checks for English naughty words and not much else. People can have fun in lots of other languages, so it would seem this is a small sample.

And specifically Italian blasphemy for some reason…

Re: The naughty username checking system used by Twitch

#67

Post by cbsmith » Thu, Oct 07, 2021, 3:15 AM UTC

woodruffw wrote: post #29

rhema wrote: post #5

I have a hard time believing this was / is the real version used. It doesn't seem broad enough. More likely it was a kind of smoketest that made sure that a more automated keyword checker was working. It does remind me of the XKEYSCORE (Snowden leaks) that used keywords to bubble up potential threats from emails etc https://www.businessinsider.com/nsa-prism-keywords-for-domes... .

Someone in another thread mentioned that these might be part of corpus generation for an ML model. That would make more sense to me.

Training an ML model to "learn" a rules engine strikes me as an incredibly bad practice. It'd make more sense to just have an actual corpus of labeled data.

Re: The naughty username checking system used by Twitch

#68

Post by monksy » Thu, Oct 07, 2021, 3:15 AM UTC

sorenjan wrote: post #55

Reminds me of the guy that streamed a talking banana on Twitch, where viewers could make it say things. People submitted variations of the n-word and got him banned, and after trying to filter out all character combinations he could think of he wrote a phonetic filter. That apparently worked much better than trying to think of every permutation of characters that sounds like bad words. https://youtu.be/bJ5ppf0po3k?t=…

That was both terrible and amazing.

Re: The naughty username checking system used by Twitch

#69

Post by PostThisTooFast » Thu, Oct 07, 2021, 3:16 AM UTC

This is one of my favorite posts on HN.

Meanwhile, I HOPE I'M NOT "POSTING TOO FAST," ASSHOLES. IT'S ONLY BEEN A COUPLE OF DAYS.

Re: The naughty username checking system used by Twitch

#70

Post by geoduck14 » Thu, Oct 07, 2021, 3:21 AM UTC

gowld wrote: post #59

rhema wrote: post #5

I have a hard time believing this was / is the real version used. It doesn't seem broad enough. More likely it was a kind of smoketest that made sure that a more automated keyword checker was working. It does remind me of the XKEYSCORE (Snowden leaks) that used keywords to bubble up potential threats from emails etc https://www.businessinsider.com/nsa-prism-keywords-for-domes... .

That list is a list of words chosen by William Knowles to taunt any NSA who may be listening. It's not a list of words used by the NSA or any spies. https://attrition.org/misc/keywords.html

In a previous life, all of our code was scanned for "vulnerabilities". One of the issues they looked for was if passwords were being stored in local variables. Initially, LOTS of people would do something like:

$Username=

$Password=

Connection.string=($Username, $Password)

The parser would flag this - Password was being stored to a variable! So we just changed our code:

$pw=. Problem solved!