You either die an MVP or live long enough to build content moderation • Hacker News Board

Re: You either die an MVP or live long enough to build content moderation

#351

Post by cmeacham98 » Wed, Sep 29, 2021, 6:22 AM UTC

0xcde4c3db wrote: post #315

bryan_w wrote: post #175

Earlier quoted context omitted.

Yup they utilize "residential proxies" to hide their behavior which makes the situation worse because sometimes it will affect your legit users

I wasn't familiar with the term, so I just searched the phrase "residential proxies", and the space seems even more sketchy than the usual VPN peddlers who promise the world and more. Are they using malware-infected PCs, or what?

The most legit ones use their own users (to access the residential proxy you have to agree to become an endpoint yourself).

More shady but probably legal ones use JavaScript and Apps that typically hide the fact you are becoming an endpoint in the T&Vs.

Straight up illegal ones use compromised computers and routers like you'd think.

Re: You either die an MVP or live long enough to build content moderation

#352

Post by New_California » Wed, Sep 29, 2021, 6:44 AM UTC

mrkurt wrote: post #160

Nextgrid wrote: post #86

Earlier quoted context omitted.

Are you doing this because of resource usage? If resources are free then you could even actually deploy their app and either whitelist it for their own IP or only allow very few requests before taking it down. This would be even more frustrating and could ruin whatever they plan to do with their abusive app in the first place. Let's say they deploy their malware/phishing page, test it a couple of times (possibly from…

We're primarily trying to prevent fraudulent payments combined with expensive VMs. Throttling CPU to almost nothing on high risk accounts sounds delightfully irritating. We also get the less resource intensive, but still harmful abusive apps that port scan the internet. Those are relatively easy to detect. We generally don't want to be a source of port scans so we shut them off pretty quickly.

How do you know the payment was fraudulent? Is there any possibility it can affect honest users?

Re: You either die an MVP or live long enough to build content moderation

#353

Post by hnbad » Wed, Sep 29, 2021, 6:47 AM UTC

I think this article glosses over some big issues with user generated content: auto-moderation is extremely fickle, humans are expensive and if your platform is large enough people will upload horrifying stuff.

There have been several stories about the poor treatment of content moderators by companies like Google (YouTube) and Facebook but to reiterate: the work can be literally traumatizing. Content moderators will be exposed to gore, child abuse and worse, frequently.

If your startup will need content moderation, think about how you will tackle it and consider the human cost of your solution. There's a trend in tech to either use AI or "fake" AI by using underpaid contractors with minimal labor rights (often in low wage countries). If your content moderation will run on humans, consider the toll and make sure they are compensated adequately and have access to mental health aid. Or better yet: consider whether you need to support massive scale user generated content to begin with. Growth isn't everything.

Re: You either die an MVP or live long enough to build content moderation

#354

Post by New_California » Wed, Sep 29, 2021, 6:51 AM UTC

mrkurt wrote: post #74

tdeck wrote: post #58

Earlier quoted context omitted.

For adversarial problems like this, a shadowban approach can sometimes be necessary. Perhaps people can still see their blogs but GoogleBot gets blocked from indexing them, or they only appear to someone with the spammer's cookies. That way it takes them longer to catch on and evade the model. Of course, that means you'll need to at least spot check your bans because you can't rely on legit users escalating to you.

We shadow ban abusive users on Fly.io and it works great. Everything seems to work right up until they try to connect to a deployed app. It took me a while to realize that ramping up the frustration level is, itself, a helpful deterrent.

Is there any possibility user is unaware she is "abusive"?

For example, I quickly searched the ToS for port scanning and didn't find it banned, yet you admit to intentionally frustrate such users.

I am not Fly.io customer at this time but I do port-scan my own services as a realtime security check.

As otherwise great service, you may open yourself for very angry user reviews if the tactic is "frustrate the user".

Re: You either die an MVP or live long enough to build content moderation

#355

Post by hnbad » Wed, Sep 29, 2021, 6:57 AM UTC

Goronmon wrote: post #100

jonshariat wrote: post #22

I'll never forget having to be a moderator for a somewhat popular forum back in the day and oh man did I learn how a few people can make your life hell. One thing not mentioned many times in these discussions are the poor moderators. Having to look at all that stuff, some of which can be very disturbing or shocking (think death, gore, etc as well as the racy things) really takes a toll on the mind. The more automatio…

I'll never forget having to be a moderator for a somewhat popular forum back in the day and oh man did I learn how a few people can make your life hell. I was also a mod for a popular gaming forum way back in the day. It was pretty miserable looking back. Personally, for me, the extreme/shocking content wasn't the biggest issue. That stuff was quick and easy to deal with. If you saw that type of content you just imme…

> What was a lot harder were the toxic users that just stuck around. Not doing anything bad enough to necessarily warrant a permanent ban, but just a constant stream of shitty behavior. Especially sometimes when the most toxic users were also some of the most popular users.

This is the problem with community guidelines being the be-all and end-all. Hard rules are great for catching insults or slurs. They're not so great for dealing with actual abuse or inciting very bad ideas.

I think Innuendo Studios' video on this problem is one of the best explanations: https://www.youtube.com/watch?v=P55t6eryY3g

There's a reason white supremacists and literal nazis (yes, really with the salutes, genocide fantasies, Jewish conspiracy theory and all) have shifted from using obvious language to dogwhistles and "just asking questions". Erosion is a much more powerful force than a few direct impacts.

If you want to moderate a community, you need to have a plan for dealing with toxic individuals, not just language. We tend to imagine "hackers" and (foul-mouthed) "trolls" but I find Molly's archetypes a lot more though-provoking: https://twitter.com/mollyclare/status/1254886822779502593?la...

I think community moderation is a problem we tend to run into the Dunning-Kruger effect with because it seems like something we have an intuitive understanding of even if we have zero experience actually doing it and having ever learned what works and what doesn't.

Re: You either die an MVP or live long enough to build content moderation

#356

Post by andrewflnr » Wed, Sep 29, 2021, 7:01 AM UTC

dcow wrote: post #272

fragmede wrote: post #254

Earlier quoted context omitted.

Maybe when the Internet was new. But whether you count the Internet's birth in the 1980's with the original cross-content and cross-country links, or around the first dot com boom and bust in 2001, or with the iPhone in 2007, we know how "the Internet" is going to interact with "the community". We knew this back in 2016 when Microsoft released their "AI" chatbot to Twitter, and Twitter taught it to be a racist asshol…

If your blog required a “real ID” to post content rather than allowed anonymous comments, would we have the same problem? The premise of the GP (and one I share) is that the internet’s content moderation problems are symptoms of default anonymity . Twitter is default anonymous so nobody’s reputation is a stake when they teach a neural net hooked up to the twitter firehose to be a racist asshole.

Facebook comments can still be a tire fire. It's not the anonymity, it's the lack of consequences.

Re: You either die an MVP or live long enough to build content moderation

#357

Post by hnbad » Wed, Sep 29, 2021, 7:05 AM UTC

Animats wrote: post #216

Roblox is working on a "moderation" system that can ban a user within 100ms after saying a bad word in voice. But their average user is 13 years old. Interestingly, Second Life, the virtual world built of user-created content, does not have this problem. Second Life has real estate with strong property rights. Property owners can eject or ban people from their own property. So moderation, such as it is, is the respon…

> Second Life has real estate with strong property rights.

Property rights is the wrong framing. We know this from the social sciences. The actual solution is just ownership.

This is the problem with public spaces in big cities compared to close-knit smaller communities: if it "belongs to everyone" it doesn't belong to anyone, if it is owned by the city/state/government, it's not actually owned by the people.

I'm not saying framing it as individual property rights doesn't work, I'm just saying it's too narrow an interpretation and a bad analogy, e.g. because property rights can allow for layers of indirection which erode this effect whereas ownership can be shared and still maintain the effect (although a cynic would then frame it as "peer pressure").

Re: You either die an MVP or live long enough to build content moderation

#358

Post by hnbad » Wed, Sep 29, 2021, 7:19 AM UTC

Sohcahtoa82 wrote: post #189

rtkwe wrote: post #71

It's really funny every time a "we don't censor" platform pops up catering too the American right they speed run going from moderation == censorship to we're moderating our platform in record time. Turns out moderation is really important to make a platform for a community.

If you create a platform with absolutely zero censorship, you become a repository for child porn. I participated in Freenet many years ago because I liked its ideas (And thought it would have been a nice way to pirate games without my ISP being able to know), but it got a reputation for being used for CP, and I promptly deleted it, because I want no part in that. If you merely censor illegal content, you will become…

I think the more general rule is that if your platform is advertising itself as an alternative to an existing dominant platform but that doesn't censor "X", the kind of people who'll flock to you will mostly be those who value X over any other interaction on the dominant platform.

Most moderates don't feel restricted enough in their speech to use Twitter even if they may have some racist ideas they don't feel safe stating there, but literal white supremacists who want to talk about the superiority of the white race all day long will love your alternative that doesn't censor them.

But if those white supremacists now all flock to your platform you likely now have a platform mostly consisting of white supremacists. This will naturally limit who else wants to join your platform because if they don't like being around tons of white supremacists they don't stick around long enough to build a counterbalance.

This can also happen after you think you've already established a more nuanced community as with your example of Freenet.

It turns out that while some people may seek opportunities to talk about X, others will not want to share space with discussions about X, often because of what it means to their own safety but also sometimes just from fear of association.

If you have a fun social network and it has a prospering community of nazis in it, you're now the nazi network, no matter how much of your community is about other things than being a nazi.

Re: You either die an MVP or live long enough to build content moderation

#359

Post by golergka » Wed, Sep 29, 2021, 8:00 AM UTC

One old story I remember is about how Lego have built their own virtual world platform and then spent enormous amount of developer time figuring out how to programmatically detect when a user-created content turned out to be shaped like a penis.

Re: You either die an MVP or live long enough to build content moderation

#360

Post by pjc50 » Wed, Sep 29, 2021, 8:11 AM UTC

evandwight wrote: post #308

michaelpb wrote: post #73

Earlier quoted context omitted.

Yeah, there's an entire category of "idea guys" who don't get this. They repeatedly try to crack the code on a truly moderation-free or purely crowd-moderated platform, and it never, ever, ever works. It almost always boils down to a poor understanding of how humans work (usually some sort of "homo economicus") or how computers work (usually some sort of "AI magic wand").

Why can't you verify real identities? Can't you use phone numbers, photos of IDs, charging a credit card, verification of physical addresses, or invites to increase the difficulty if creating fake accounts? Yes, there is an issue of increasing the difficulty if signing up for real users but once accounts are tied to real identities doesn't that allow crowd moderation? - person whose fallen into your apparent fallacy

That swings to the other side of "you either die an MVP or build content moderation": people are not going to submit real ID for a random project. They've only just started implementing this on Youtube ""age verification"" because they were made to, and Facebook only did it as an arbitrary after-the-fact hammer. It causes all sorts of problems (what do you regard as valid? What about deadnames?).

Twitter and lots of other sites do phone number verification which is less onerous but far easier to spoof.

And of course the biggest, highest profile moderation challenge involves people whose identities are known but nonetheless are toxic to the community. Including the "final boss" of content moderation challenges, Donald Trump.