Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You either die an MVP or live long enough to build content moderation

mux.com

Post reply on HN
Print view View original

51–60 of 429 posts

Re: You either die an MVP or live long enough to build content moderation

#51

Post by Alir3z4 »

YetAnotherNick wrote: post #48
Alir3z4 wrote: post #17

Around 1 year ago we got hit badly on our [blogging platform][0] by people/groups submitting fake customer support description of other big companies, either being Microsoft, Facebook, Comcast etc. Rolled out a machine learning model and trained it on the database. 99% of them vanished. Next day, the machine didn't work and success rate was around 5%. Found out, they have learned the trick and now using symbols from…

How do you detect the ground truth for training the model? Do you manually label it?

Yes, simple classification. Nothing fancy.

Basically, pulled the database into CSV file and anything that was published before the bad content was classified as HAM.

We had content that were OK, so marked as HAM and then our new bad content all marked as SPAM.

When deployed to production for some hours HAM content got wrongly marked and model got trained on them as well which made so many confusion but the problem was taken care of once the model got properly tuned and safer to let it be automated.

Re: You either die an MVP or live long enough to build content moderation

#52

Post by mooreds »

I agree. My employer has a moderation product (for comments, usernames, etc): https://cleanspeak.com/

I don't work with it much, but from what I can see it's surprisingly complicated to filter out comments quickly without impacting user experience. I guess you know you've succeeded when the pottymouths join your platform :).

Re: You either die an MVP or live long enough to build content moderation

#53

Post by seany »

cowmoo728 wrote: post #29
seany wrote: post #4

Freenet/lbry/tor hidden sites all exists (and get used all the time) and it's 100% not required there at all. I hope at some point weird moralization of nudity will stop.

Have you gone on darknet sites? They have moderation too, or else they get filled with CP and terrorist propaganda just like every other service. I guess that's "fine" if you're anonymous and don't think the FBI will find you. But if you're running a business on the clearnet there's a real name and address and there will be real life consequences. The FBI gets interested real fast if you don't moderate posts that enc…

Yes, near daily.

Re: You either die an MVP or live long enough to build content moderation

#54

Post by jffry »

Y_Y wrote: post #25
Alir3z4 wrote: post #17

Around 1 year ago we got hit badly on our [blogging platform][0] by people/groups submitting fake customer support description of other big companies, either being Microsoft, Facebook, Comcast etc. Rolled out a machine learning model and trained it on the database. 99% of them vanished. Next day, the machine didn't work and success rate was around 5%. Found out, they have learned the trick and now using symbols from…

If your eyes can "normalize" a unusual symbols to a common one to make an English word then so can a lookup table. I feel like this isn't a case where you'd reach first for a neural net.

In fact, the Unicode consortium provides a report and extensive list of "confusable" symbols, which you could use alongside Unicode normalization tables to map adversarial back into more ASCII-equivalent text before running it through anti-spam mechanisms that are interested in the content of the message.

https://www.unicode.org/reports/tr39/

Re: You either die an MVP or live long enough to build content moderation

#55

Post by jfengel »

If you host blobs for free, somebody is going to use you as their host. Even if you just hosted audio, I'm sure somebody will quickly come along with a steganography tool to hide their content on your site (and use your bandwidth).

Similarly, if you make compute power available, people will use you to mine cryptocurrency. Even if all you host is text, somebody will come along to be abusive. When you put a computer on the Internet, it's open to the entire world, including the very worst people.

If you're hosting a community, start from the beginning by knowing who your community is and how they will tell you who they are. If the answer is "everybody", then know what everybody means -- it means some people won't want to be there, because some people will make life hard for them.

It's no longer 1991, when you could assume that such people wouldn't find you. They will find you -- for money, or the lulz. You have to plan for that on day 1. You can't fix it after the fact.

Re: You either die an MVP or live long enough to build content moderation

#56

Post by jjulius »

quantumBerry wrote: post #36
majormajor wrote: post #24

Earlier quoted context omitted.

The internet is very moderated, on the contrary, in terms of UGC. Traditional, non-social, websites have single or known-group authors. When one of them is defaced or modified we call it "hacking" not "unmoderated content." We assume NASA's site has NASA-posted content. We assume Apple's site has Apple-posted content. Sites with different standards for what they'd publish have been around for decades (for gore, for p…

The internet is not moderated in any useful sense for content. Drug markets like white house market, and before that silk road have perpetuated for years. Tor and other darknet websites host content that is nearly universally disdained by governments and even most individuals, which I hesitate to even name here what that heinous content is (you and I both know some examples). > We assume NASA's site has NASA-posted c…

>The internet is not moderated in any useful sense for content. Drug markets like white house market, and before that silk road...

You mean the Silk Road that the US government "moderated" out of existence, along with other Tor marketplaces over the years? The same ones that suggest White House Market's existence is also likely to be limited?

Re: You either die an MVP or live long enough to build content moderation

#57

Post by jffry »

Alir3z4 wrote: post #49
Y_Y wrote: post #25

Earlier quoted context omitted.

If your eyes can "normalize" a unusual symbols to a common one to make an English word then so can a lookup table. I feel like this isn't a case where you'd reach first for a neural net.

Yeah, then someone has to create or find that whole table and make. The initial problem wasn't those symbols but the content itself, the symbols and special characters came into the problem later. Later on as mentioned in my original comment, that they would use positive content from other blog posts that were published/passed the moderation to mix up their bad content. Probably could use a different method, but at t…

The good news is that the Unicode consortium has a report on this issue, and the tables already exist for normalization and mapping of confusables to their ASCII lookalikes: https://www.unicode.org/reports/tr39/

Re: You either die an MVP or live long enough to build content moderation

#58

Post by tdeck »

Alir3z4 wrote: post #17

Around 1 year ago we got hit badly on our [blogging platform][0] by people/groups submitting fake customer support description of other big companies, either being Microsoft, Facebook, Comcast etc. Rolled out a machine learning model and trained it on the database. 99% of them vanished. Next day, the machine didn't work and success rate was around 5%. Found out, they have learned the trick and now using symbols from…

For adversarial problems like this, a shadowban approach can sometimes be necessary. Perhaps people can still see their blogs but GoogleBot gets blocked from indexing them, or they only appear to someone with the spammer's cookies. That way it takes them longer to catch on and evade the model.

Of course, that means you'll need to at least spot check your bans because you can't rely on legit users escalating to you.

Re: You either die an MVP or live long enough to build content moderation

#59

Post by pbreit »

Alir3z4 wrote: post #17

Around 1 year ago we got hit badly on our [blogging platform][0] by people/groups submitting fake customer support description of other big companies, either being Microsoft, Facebook, Comcast etc. Rolled out a machine learning model and trained it on the database. 99% of them vanished. Next day, the machine didn't work and success rate was around 5%. Found out, they have learned the trick and now using symbols from…

How did the lousy content affect your legit users?

Re: You either die an MVP or live long enough to build content moderation

#60

Post by Alir3z4 »

cowmoo728 wrote: post #35
Alir3z4 wrote: post #17

Around 1 year ago we got hit badly on our [blogging platform][0] by people/groups submitting fake customer support description of other big companies, either being Microsoft, Facebook, Comcast etc. Rolled out a machine learning model and trained it on the database. 99% of them vanished. Next day, the machine didn't work and success rate was around 5%. Found out, they have learned the trick and now using symbols from…

I'm interested in why these people were doing this. Were they hoping to get non-tech-savvy people that were searching for computer help? I guess that's a good audience of unwitting users to attempt to hack, but was the goal to get them to submit to one of the remote tech support scams? Were they embedding malware into your blogging platform, or getting ad revenue out of this somehow?

> Were they hoping to get non-tech-savvy people that were searching for computer help?

Yes.

They would create this posts and get quickly on search results (The platform is pretty good for making SEO optimization out of the box) and they would write good quality posts as well.

They would also share this posts on some other websites, especially social media accounts.

We don't have Google analytics or such to see where exactly they would come from. I noticed huge traffic to such pages by looking at the logs.

Our nginx log parser was alerting us about sudden spike on certain blogs and pre-defined list of words we have.

That's when we noticed something is going on.

Didn't take more than couple of hours (while working on the model) that we receive email from data center people about hosting phishing content, again didn't take much longer we received emails from some of those companies as well.

> Were they embedding malware into your blogging platform, or getting ad revenue out of this somehow?

No. On the blogging platform, we have everything bleached out, nothing would go in without passing through sanitizers.

They would simply had people convinced to call those US numbers.

I actually called one of those numbers and yeah, it was one of those customer supports some other part of the planet earth and definitely not from the company he was pretending to be and very quickly asked me to install team viewer on my machine. I really wanted to let them access it via the windows on my virtual box and have some fun with them, but well, someone had to fix the moderation issue :D

Post reply on HN