Live data from Hacker News

The naughty username checking system used by Twitch

ghostbin.com

91–100 of 348 posts

Re: The naughty username checking system used by Twitch

#91
I wonder why they didn't go with syllables and something like a levenshtein distance to syllables?

That way they could more easily maintain similar looking and sounding words, including leetspeak and other variants.

Because with this kind of approach, something like "yolocaust" will get through, as most checks only go with exact matches and permutations will always get around it easily...

Whereas with something like a levenshtein distance you could compare it with a set of words and syllables and if it's too similar looking, e.g. 90% the same distance compared to username length, you could simply block it.

Re: The naughty username checking system used by Twitch

#93

We had to do this for a link shortening system (to make sure random base64 didn't contain profanity). It was a pretty fun problem. Not just the implementation, but doing the math to make sure it didn't make our shortened links easily enumerable. The implementation wasn't too bad, but we set up logging initially to spit out any random strings it decided to block. I demo'd this in front of the whole company and live ta…

> to make sure random base64 didn't contain profanity

I would have said "why bother" until this happened to us.

A customer rang us up in a fury because some demo/ random data that we generated happened to have the word "penis" in it. They were convinced we must have put it there because we thought he was a cock. It was very difficult to defuse the situation.

Re: The naughty username checking system used by Twitch

#94
post #11
post #5

I have a hard time believing this was / is the real version used. It doesn't seem broad enough. More likely it was a kind of smoketest that made sure that a more automated keyword checker was working. It does remind me of the XKEYSCORE (Snowden leaks) that used keywords to bubble up potential threats from emails etc https://www.businessinsider.com/nsa-prism-keywords-for-domes... .

It also mostly checks for English naughty words and not much else. People can have fun in lots of other languages, so it would seem this is a small sample.

After looking through it quickly it seems to do the same as most profanity checks with Dutch: it doesn't block "kut" (a crude word for female genitalia) but it does block "kunt" (third person form of "can")

Re: The naughty username checking system used by Twitch

#95
post #18

Whats this one about? CREATE OR REPLACE FUNCTION is_blasphemy (VARCHAR) RETURNS BOOLEAN STABLE AS $$ SELECT replace($1,'_','') SIMILAR TO '%p(o|0)rc(o|0)di(o|0)%' OR replace($1,'_','') SIMILAR TO '%p(o|0)rc(o|0)mad(o|0)nna%' $$ LANGUAGE SQL;

That one was my favorite too. Filter lists don't work well in a multi lingual world. https://www.reddit.com/r/Rainbow6/comments/a01w7q/got_banned...

Re: The naughty username checking system used by Twitch

#96

We had to do this for a link shortening system (to make sure random base64 didn't contain profanity). It was a pretty fun problem. Not just the implementation, but doing the math to make sure it didn't make our shortened links easily enumerable. The implementation wasn't too bad, but we set up logging initially to spit out any random strings it decided to block. I demo'd this in front of the whole company and live ta…

When it comes to censoring randomly generated strings, I like simply to omit vowels from the alphabet. Usually I'll omit some of the more obvious lookalikes too, e.g. [1 0 v]. It's a simple solution. Sure, it is still possible for something to slip through that looks similar to something bad. But the potential to strongly offend is greatly reduced.

Gov.uk did this. The code WNKR still ended up on Reddit yesterday.

Re: The naughty username checking system used by Twitch

#97

Earlier quoted context omitted.

It's an endless arms race. I spend a lot of time studying unsavory people and a great deal of effort goes into the development of new dogwhistles that are designed to either provoke or connect with peers while maintaining deniability. You might like this paper on the evolutionary dynamics of covert social signaling: https://www.nature.com/articles/s41598-018-22926-1

If they're covert and only members of the group using them know what they mean, then why fight it? Seems like a helpful way to communicate about things that others might not like to hear. The paper you linked said that too - "Such signals may allow coordination and enhanced cooperation while also avoiding the alienation or hostile reactions of individuals with different preferences.". This happens all the time in car…

Because it encourages more of the same, and it's rarely completely covert. Often and increasingly, it's racial dogwhistling and just devolves every thread to wasteful, antagonistic and unproductive conversation. You might've heard the line/story about accommodating nazis in your bar.

Re: The naughty username checking system used by Twitch

#98
post #12

If you liked this chaos, you'd love my 15+ years of cobbled-together efforts at limiting forum spam and the like. I suddenly don't feel quite so alone in the myriad efforts needed to tackle this sort of thing.

At an online company I worked at we had various word filters for our forums, but new stuff was always popping up and getting through. What worked in the end was having any newly created thread send a message containing the post title & body to a slack channel specifically for monitoring the forums. Employees and our forum moderators were in there, and any bad threads were nearly instantly deleted. Eventually the spam…

I get alerted to every new thread, and when I'm online my response rate is also very fast. When the message count was under 1,000/week, I used to get an email for every single post too. But if other moderators aren't around and I'm offline, I am out of luck. Shadow-banning can be effective. I also give regulars the ability to sin-bin any post which removes it from view and leaves it for me to check.

Having a dozen reviewers, especially if spread across timezones, would be a dream!

Re: The naughty username checking system used by Twitch

#99

Earlier quoted context omitted.

SMS is the biggest constraint. Unicode characters trigger lower segment char limits (effectively doubling the cost of a 71 char text message). And also it's important that the links can be clicked on a smartphone. So url-safe base64 (some shorteners use base62). And numbers can be N4u6hty too, so you gotta catch those cases.

Goodness the scope of the problem just exploded in my mind after this explanation.

The good news is that things like https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and... exist so getting a source of words to filter is easy enough. And converting numbers to letters isn't too bad.

The hardest problem with the implementation was that with a long list you can't just search for a few dozen inappropriate words (like the Twitch implementation). It would be very expensive to do hundreds or even thousands of checks against every inappropriate word.

The solution we came to was to truncate all the inappropriate words to either 3 or 4 letters and store them in a big set. We then take our generated strings, which are usually 11 characters, and break them up into all possible substrings of lengths 3 and 4. For example, 1a2b3c4d5e6 would be broken down into 1a2 a2b 2b3 b3c 3c4 c4d 4d5 5e6 1a2b a2b3 2b3c b3c4 3c4d c4d5 4d5e d5e6. An 11 character string would always have 16 such substrings. We then check all 16 against the banned set. 16 lookups into a set is pretty cheap and as we have expanded the word set over time (e.g. add a new language) our performance hasn't changed.

One drawback to our approach is that we do have false positives but we did the math and our space was still large enough, the cost of generating a new one was pretty low, and customers never see it so it's just not a big deal to throw out false positives.

Re: The naughty username checking system used by Twitch

#100
post #71

> LIKE '%aggin%' Looks like "Baggins" is banned. Poor Bilbo...

Sure, but...

The story we know was written by him. I wonder what the trolls would say, or the dead dragon, or the town that his actions helped destroy?

Is he a hero, or, just the guy who wrote it all down?

And just when something really important happens, he throws a powerful weapon(the one right) at his nephew and goes away to retire!

A life lead with riches (gold from the trolls), a ring granting him extremely long life and health, and yup.. off he goes, first sign of real trouble.

Poor Bilbo indeed!

Post reply on HN