Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You either die an MVP or live long enough to build content moderation

mux.com

Post reply on HN
Print view View original

221–230 of 429 posts

Re: You either die an MVP or live long enough to build content moderation

#221

Post by andrei_says_ »

tdeck wrote: post #58
Alir3z4 wrote: post #17

Around 1 year ago we got hit badly on our [blogging platform][0] by people/groups submitting fake customer support description of other big companies, either being Microsoft, Facebook, Comcast etc. Rolled out a machine learning model and trained it on the database. 99% of them vanished. Next day, the machine didn't work and success rate was around 5%. Found out, they have learned the trick and now using symbols from…

For adversarial problems like this, a shadowban approach can sometimes be necessary. Perhaps people can still see their blogs but GoogleBot gets blocked from indexing them, or they only appear to someone with the spammer's cookies. That way it takes them longer to catch on and evade the model. Of course, that means you'll need to at least spot check your bans because you can't rely on legit users escalating to you.

I’d presume that large scale spammers check their work in incognito via a different network. It’s their job.

Re: You either die an MVP or live long enough to build content moderation

#222

Post by PeterisP »

throwawaysea wrote: post #213
Sohcahtoa82 wrote: post #189

Earlier quoted context omitted.

If you create a platform with absolutely zero censorship, you become a repository for child porn. I participated in Freenet many years ago because I liked its ideas (And thought it would have been a nice way to pirate games without my ISP being able to know), but it got a reputation for being used for CP, and I promptly deleted it, because I want no part in that. If you merely censor illegal content, you will become…

> If you merely censor illegal content, you will become a home for disinformation and ultra right-wing conspiracies. See Parler. What do you count as disinformation and why is it a problem? If you disagree with something you can ignore it and move on, or engage with it and respond with your own counter-argument. It doesn't seem like a problem that reduces the viability of the entire platform. It is also strange to me…

It is a problem because you'll get thrown out by your hosting and other service providers if you don't moderate your content; so if you want to keep running your service, not moderating is simply not a practical option. That is why Parler is mentioned, they are a demonstration that it's not practical to keep operating without accepting a duty to moderate (as Parler did eventually) even if you try really, really hard.

And while there are a lot of conspiracies, all of which will be on your site if you don't moderate, most of them will be tolerated by others but it's the ultra right-wing conspiracies / nazis / holocaust deniers that will cause your service threats of disconnection; so you'll either start moderating or get your service killed in order to protect them.

I understand you don't want anyone - whether the government, powerful private companies, or biased moderators - to become the arbiters of permitted opinions; however, you don't really get to choose (and neither do I); currently there are de facto arbiters in this world.

Re: You either die an MVP or live long enough to build content moderation

#223

Post by wanderingstan »

Alir3z4 wrote: post #62
jffry wrote: post #57

Earlier quoted context omitted.

The good news is that the Unicode consortium has a report on this issue, and the tables already exist for normalization and mapping of confusables to their ASCII lookalikes: https://www.unicode.org/reports/tr39/

Oh, that's nice. I guess I can use that next time time to work on the data cleaning for that model. Thanks.

As I commented above:

I built a Python library for finding strings obfuscated this way. Was critical when moderating our telegram channel before an ICO. https://github.com/wanderingstan/Confusables E.g. "𝓗℮𝐥1೦" would match "Hello"

Re: You either die an MVP or live long enough to build content moderation

#224

Post by Nasrudith »

phkahler wrote: post #133

Here's a thought. Have a platform where identity is verified. Users can post publicly or within their circle. Any illegal or fraudulent content can be handled by the legal system due to the lack of anonymity. Beyond that, let users form groups for topics like reddit. Where does this fall down?

We tried that with "real ID" policies. It just made people commit to their shittiness openly. Not to mention they always have repudiation. Even if we go with full fledged cryptography the opsec will fail at scale.

Re: You either die an MVP or live long enough to build content moderation

#225

Post by Spone »

bcrosby95 wrote: post #171
Alir3z4 wrote: post #17

Around 1 year ago we got hit badly on our [blogging platform][0] by people/groups submitting fake customer support description of other big companies, either being Microsoft, Facebook, Comcast etc. Rolled out a machine learning model and trained it on the database. 99% of them vanished. Next day, the machine didn't work and success rate was around 5%. Found out, they have learned the trick and now using symbols from…

When we had this problem, we added an input element positioned off-screen and ignored submissions that populated it. Cleaned it all up.

This is usually called a "honey pot" if some people want to search tools for implementing it.

Re: You either die an MVP or live long enough to build content moderation

#226

Post by Nasrudith »

munificent wrote: post #33
quantumBerry wrote: post #7

The internet itself is unmoderated in any useful sense for content, yet it has lived longer than most of these cheesy "moderated" products that seek to impose their morality on you.

It looks like you're getting downvoted, but I think this is a good point and worth thinking about. I believe one key difference here is group identity perception . If you like thinking in business terms, you could say "branding". Facebook, Reddit, HN, Twitter, etc. all must care about content moderation because there is a feedback loop they have to worry about: 1. Toxic content gets posted. 2. Users who dislike that…

I personally see it as "plausible deniability" as the cynical actual distinction for what gets people to share blame. Not actual affiliations or whose servers it is run on. Any number of objectionable sites are run on AWS and you basically need to be an international scandal or violating preexisting terms to get booted. Like some malware to governments merchants. Amazon's policies did not care if it was legal just if you were doing so unauthorized. A wise move when international law is really like the Pirate code.

The interlinking between the pages themselves and common branding are what creates the associations. Distributed twitter alternatives like Mastodon can even share the same branding but it is on a per network basis and complex enough to allow for some "innocent" questionable connections.

Re: You either die an MVP or live long enough to build content moderation

#227

Post by rtkwe »

User23 wrote: post #93
rtkwe wrote: post #71

It's really funny every time a "we don't censor" platform pops up catering too the American right they speed run going from moderation == censorship to we're moderating our platform in record time. Turns out moderation is really important to make a platform for a community.

How is that funny? Every platform has to block illegal content. Every platform wants to block low value content like spam. Many platforms want to block obscenity and pornography. None of this is in any way news to any of the platforms you’re alluding to. The interesting distinction between platforms is not whether they moderate, but what lawful and non-abusive (of the platform itself) content they permit. Edit: Child…

It's not illegal content I'm talking about. I'm specifically thinking of sites like Parler and Gab which very loudly and specifically start as anti-censorship of anything legal for the American right that feel like they're being censored off of the regular platforms. Then they quickly learn that no moderation means you'll get absolutely flooded with trolls who aren't a fan of your chosen ideology and are willing to spam and troll you. That quickly encourages them to start actually moderating in some regards the exact thing they were created in opposition to because what they're actually mad about is specific moderation decisions not the idea of moderation in general.

Re: You either die an MVP or live long enough to build content moderation

#228

Post by kbenson »

Alir3z4 wrote: post #178
jart wrote: post #167

Earlier quoted context omitted.

Did they map to ISP ASNs? Country geolocation doesn't say much anymore since there's so many VPN providers whose business is to buy a CIDR in every country and resell access.

Yes, almost all of them mapped to ISP ASNs. Very few of them were from AWS, OVH and other hosting providers, very very few. We ran each IP towards black lists ips, paid IP reputation checkers. Majority of the IPs were clean. Back then we had IP reputation check, but it was a headache to maintain, so we disabled it later, however that time very very few them got stopped at IP reputation checks.

As someone that's used IP proxying services that provide millions of IPs for scraping purposes, that is a very mature industry, and they advertise (and I believe them) "millions" or IPS, even for what you might consider hard to supply ones, like mobile IPs, and they let you slice and dice them however you want? Datacenter IPs? Residential IPs? Mobile IPs?[1] What state or city would you like them in? Would you like the site you're hitting to not have been accessed by this IP (through proxying at least), and if so how many days? Do you want some mix of that? Make your own configurations and set them up as proxy endpoints, etc.

Fighting against abuse at the level of IP address attributes seems like a losing game to me. Honestly, the best I saw at this (3-5 years ago at least) for traffic was Distil networks, where they put a proxy device in front and examine your traffic and captcha or block based on that.

Since you have content being submitted, there's a lot more you can use to classify, such as how you used ML, so that's good. Part of me worries that this is all sort of reminiscent of infections and antibiotics though. The continual back-and-forth of you finding a block them finding a workaround feels kind of like you were training the spammers (even if you were training yourself at the same time). At some point maybe we'll find that most the forum spam is ML generated low information content posts that also happen to be astroturfing that is hard to distinguish from real people's opinions.

1: Fun fact, to my knowledge anonymous mobile IPs are provided by a bunch of apps opting into an SDK (like an advertising/metrics SDK) which while their app is open (at least I hope that's a requirement) registers itself to the proxying service so it can be handed out for use by paying proxy customers. Think about that next time you play your free "ad-supported" mobile game.

Re: You either die an MVP or live long enough to build content moderation

#229

Post by dexwiz »

Animats wrote: post #216

Roblox is working on a "moderation" system that can ban a user within 100ms after saying a bad word in voice. But their average user is 13 years old. Interestingly, Second Life, the virtual world built of user-created content, does not have this problem. Second Life has real estate with strong property rights. Property owners can eject or ban people from their own property. So moderation, such as it is, is the respon…

For some reason I remember everyone's behavior on old school message boards as much better than modern social media. Sure, you have your degenerate boards, but just don't go there. Moderation and censor ship will always exist, but they seem to work better when they are more locally applied.

Re: You either die an MVP or live long enough to build content moderation

#230

Post by flal_ »

Alir3z4 wrote: post #44
vadfa wrote: post #23

Earlier quoted context omitted.

What software/libraries have you used for your machine learning moderation system?

tensorflow and scikit-learn to train and build the model. On the front FastAPI (behind uvicorn) to accept calls via REST API. Deployed via docker. To be honest, tensorflow and scikit-learn may not be the right fit for everything. Every situation needs different approach and different solution. Worth nothing, the most time consuming part was dealing with data itself and not model training or machine learning. In coupl…

"Worth nothing, the most time consuming part was dealing with data itself and not model training or machine learning."

Become a data scientist they said. Yay, artificial intelligence...

Post reply on HN