It's really funny every time a "we don't censor" platform pops up catering too the American right they speed run going from moderation == censorship to we're moderating our platform in record time. Turns out moderation is really important to make a platform for a community.
You either die an MVP or live long enough to build content moderation
71–80 of 429 posts
Re: You either die an MVP or live long enough to build content moderation
#72Hey there :). Author here. It was a fun experiment to play around with some different strategies for adding content moderation to https://stream.new Hive ended up being the one I landed on after trying Google Vision first ( https://cloud.google.com/vision ). The other one I was looking at is Clarity.ai but I didn't get a chance to try that one yet.
One thing that wasn't obvious to me is, why did you care about uploads of NSFW? As I understood it, you want to become Imgur of video. Imgur only became so big because they allowed NSFW stuff.
Re: You either die an MVP or live long enough to build content moderation
#73If you host blobs for free, somebody is going to use you as their host. Even if you just hosted audio, I'm sure somebody will quickly come along with a steganography tool to hide their content on your site (and use your bandwidth). Similarly, if you make compute power available, people will use you to mine cryptocurrency. Even if all you host is text, somebody will come along to be abusive. When you put a computer on…
Yeah, there's an entire category of "idea guys" who don't get this. They repeatedly try to crack the code on a truly moderation-free or purely crowd-moderated platform, and it never, ever, ever works.
It almost always boils down to a poor understanding of how humans work (usually some sort of "homo economicus") or how computers work (usually some sort of "AI magic wand").
Re: You either die an MVP or live long enough to build content moderation
#74Around 1 year ago we got hit badly on our [blogging platform][0] by people/groups submitting fake customer support description of other big companies, either being Microsoft, Facebook, Comcast etc. Rolled out a machine learning model and trained it on the database. 99% of them vanished. Next day, the machine didn't work and success rate was around 5%. Found out, they have learned the trick and now using symbols from…
For adversarial problems like this, a shadowban approach can sometimes be necessary. Perhaps people can still see their blogs but GoogleBot gets blocked from indexing them, or they only appear to someone with the spammer's cookies. That way it takes them longer to catch on and evade the model. Of course, that means you'll need to at least spot check your bans because you can't rely on legit users escalating to you.
We shadow ban abusive users on Fly.io and it works great. Everything seems to work right up until they try to connect to a deployed app.
It took me a while to realize that ramping up the frustration level is, itself, a helpful deterrent.
Re: You either die an MVP or live long enough to build content moderation
#75Around 1 year ago we got hit badly on our [blogging platform][0] by people/groups submitting fake customer support description of other big companies, either being Microsoft, Facebook, Comcast etc. Rolled out a machine learning model and trained it on the database. 99% of them vanished. Next day, the machine didn't work and success rate was around 5%. Found out, they have learned the trick and now using symbols from…
Tale of humans man...
Re: You either die an MVP or live long enough to build content moderation
#76If you host blobs for free, somebody is going to use you as their host. Even if you just hosted audio, I'm sure somebody will quickly come along with a steganography tool to hide their content on your site (and use your bandwidth). Similarly, if you make compute power available, people will use you to mine cryptocurrency. Even if all you host is text, somebody will come along to be abusive. When you put a computer on…
> If you host blobs for free, somebody is going to use you as their host. Even if you just hosted audio, I'm sure somebody will quickly come along with a steganography tool to hide their content on your site (and use your bandwidth). This feels like something more of a theoretical example cited versus something that has happened. Do you have any examples of steganography being used as bandwidth redirection/hosting?
Re: You either die an MVP or live long enough to build content moderation
#77Around 1 year ago we got hit badly on our [blogging platform][0] by people/groups submitting fake customer support description of other big companies, either being Microsoft, Facebook, Comcast etc. Rolled out a machine learning model and trained it on the database. 99% of them vanished. Next day, the machine didn't work and success rate was around 5%. Found out, they have learned the trick and now using symbols from…
If your eyes can "normalize" a unusual symbols to a common one to make an English word then so can a lookup table. I feel like this isn't a case where you'd reach first for a neural net.
If you can identify text written with mixed glyphs just ban it outright. Normal users don't use text like this, the pure binary presence of such "homomorphic" text at all is probably a better signal for spam than whatever your neural net when running it after normalization.
Re: You either die an MVP or live long enough to build content moderation
#78Around 1 year ago we got hit badly on our [blogging platform][0] by people/groups submitting fake customer support description of other big companies, either being Microsoft, Facebook, Comcast etc. Rolled out a machine learning model and trained it on the database. 99% of them vanished. Next day, the machine didn't work and success rate was around 5%. Found out, they have learned the trick and now using symbols from…
Do you have a mechanism for appealing the automated process?
Not sure if I understand correctly, but if you mean how re-training and deployment would be.
Nothing fancy tbh.
For the first several deployment while taking care of edge case and debugging, all manually on my own laptop and shot into cluster as a docker image.
Later, when starting to classify more content on the platform itself:
- Webhook will trigger the CI to train the model with new ham and spam content. - A new docker image was created and deployed to container.
The above would fail if success rate in validation/testing would be below 95%.
Still when classifying bad content, the whole process happens automated.
Special thanks to Gitlab CI, shell scripts, Python and Docker Swarm.
Re: You either die an MVP or live long enough to build content moderation
#79Around 1 year ago we got hit badly on our [blogging platform][0] by people/groups submitting fake customer support description of other big companies, either being Microsoft, Facebook, Comcast etc. Rolled out a machine learning model and trained it on the database. 99% of them vanished. Next day, the machine didn't work and success rate was around 5%. Found out, they have learned the trick and now using symbols from…
If you don't mind me asking, what sentence embeddings model (bert/roberta/etc) did you have the best luck with for your classifier? I like the quick retrain that can be done with an approach like this, though I have found that if you throw too many different SPAM profiles at a classifier it starts to degrade, and you might have to build multiple and ensemble them. The embedding backend can help a lot with that.
Re: You either die an MVP or live long enough to build content moderation
#80If you host blobs for free, somebody is going to use you as their host. Even if you just hosted audio, I'm sure somebody will quickly come along with a steganography tool to hide their content on your site (and use your bandwidth). Similarly, if you make compute power available, people will use you to mine cryptocurrency. Even if all you host is text, somebody will come along to be abusive. When you put a computer on…
> If you host blobs for free, somebody is going to use you as their host. Even if you just hosted audio, I'm sure somebody will quickly come along with a steganography tool to hide their content on your site (and use your bandwidth). This feels like something more of a theoretical example cited versus something that has happened. Do you have any examples of steganography being used as bandwidth redirection/hosting?
Not at all theoretical, this happens all the time. There are tools like StegoShare: https://en.wikipedia.org/wiki/StegoShare
I googled and one of the top articles was this (I didn't read it): https://www.deccanchronicle.com/technology/in-other-news/070...
I couldn't find the article I had read a few years back, but I remember this sort of thing being used to host content on Facebook, Wikipedia, Reddit, etc, before they cracked down on it.