Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Perceptual hashing tools for detecting child sexual abuse material

github.com

Post reply on HN
Print view View original

31–40 of 52 posts

Re: Show HN: Perceptual hashing tools for detecting child sexual abuse material

#31

Post by faustomorales »

probablypat wrote: post #26
faustomorales wrote: post #2

Hi HN! I'm a data scientist at Thorn, a non-profit dedicated to defending children from sexual abuse. We're excited to open source some elements of our perceptual hashing tooling as a Python package. We've tried to make it very flexible both for ourselves and hopefully also for others. Our aim with is to provide tools that (1) help more people eliminate child sexual abuse material from the internet and (2) assist wit…

Thank you for what you and your team do. Is there anyway I can help contribute to your organization?

Thank you so much for asking! Naturally, part of why we wanted to share this with the broader community was to make it so interested people can jump in and help out in the open. And I would be remiss if I didn’t mention the fact that we’re hiring! [1]

We can always use help raising awareness. Advocating for more survivor resources is a great place to start. Help spread the word through your social networks by connecting with us on Facebook and Twitter. Learn about even more ways to get involved and subscribe to our newsletter for general updates [2].

[1] https://thorn.org/careers

[2] https://thorn.org/join-us

Re: Show HN: Perceptual hashing tools for detecting child sexual abuse material

#32

Post by faustomorales »

nieve wrote: post #27
faustomorales wrote: post #7

Earlier quoted context omitted.

Thanks for asking these important questions! * What are the challenges surrounding verification that your system functions properly, given that the test material is illicit? You're right that storing child sexual abuse material (CSAM) is illegal, unless you are the National Center for Missing and Exploited Children (NCMEC) or law enforcement. What is legal is to maintain a hash of known CSAM. NCMEC, Law Enforcement,…

A false positive rate of 1/1000 is hard to assess without actual prevalence stats, but with a decent-sized userbase it seems likely you're still going to get a significant number of false positives. Is it intended that users of your system would have employees manually vet all positives (with legal and mental health concerns) or just submit them without review? I'm coming from having built tools to support a large ma…

Great question — organizations decide how to handle reviews internally. So the answer to your question on “review all” versus “automatically submit” is a, perhaps unsatisfying, but honest: it depends. We provide a guide [1] to help organizations formulate their own policies. And we're currently working on a content moderation tool that focuses on helping organizations operationally handle problematic content and considers the wellness and resiliency of reviewers.

[1] https://www.thorn.org/sound-practices-guide-stopping-child-a...

Re: Show HN: Perceptual hashing tools for detecting child sexual abuse material

#33

Post by claudiawerner »

tempaccount8354 wrote: post #10

If you dare to look into this topic, please also take a moment to look to another child protection non-profit[1] that (quite unusually) considers human and civil rights and sex positivity as one of its core values. You might feel uneasy about "sex positivity" being associated with preventing child sexual abuse. They also have a lot of other messages that might initially repel you. But I also think that "when you asso…

I think it's good you posted this; the goals listed on the first link seem very reasonable to get behind, and it's something both I and researchers in child sexual abuse material (e.g. Amy Adler's paper, which is one of the most cited in the field) have noticed. The rhetoric of "think of the children" can be and has been used many times to silence sexual activity between adults, including fictional representations (such as manga and comics, some of which are illegal to possess in the UK), kinks (online ageplay roleplay is often targeted under obscenity law, despite being between two adults), and other cases.

In my view, the protection of children based on hard evidence showing causal links is paramount, not suspicion and anecdota (it was, in fact, anecdota given before Parliament which was used in the creation of the bill in England and Wales to illegalize lolicon manga) based on the "social harms" that seem obvious at first sight but are refuted by cultural anthropology (e.g Patrick Galbraith and Mark McLelland have come to very interesting conclusions regarding how adults consume fantasy material). There is no group to fight for the few thousand people caught under this law so far (according to the English VAWG report 2017), nor do I imagine their being. The organization you linked seems, at least, to have this sort of thing on the agenda.

Re: Show HN: Perceptual hashing tools for detecting child sexual abuse material

#34

Post by nieve »

faustomorales wrote: post #32
nieve wrote: post #27

Earlier quoted context omitted.

A false positive rate of 1/1000 is hard to assess without actual prevalence stats, but with a decent-sized userbase it seems likely you're still going to get a significant number of false positives. Is it intended that users of your system would have employees manually vet all positives (with legal and mental health concerns) or just submit them without review? I'm coming from having built tools to support a large ma…

Great question — organizations decide how to handle reviews internally. So the answer to your question on “review all” versus “automatically submit” is a, perhaps unsatisfying, but honest: it depends. We provide a guide [1] to help organizations formulate their own policies. And we're currently working on a content moderation tool that focuses on helping organizations operationally handle problematic content and cons…

Oh good, I'm sure most organizations can use something like that guide as well as the tools. There's a lot of legitimate worry about both the wellness side & the legal exposure issues, but it seems like beyond the common wisdom to be very careful (somehow) I think in a lot of minds there's a lack of clarity as to what exactly that means. Is there a particular reason access to the guide requires handing over contact information?

Re: Show HN: Perceptual hashing tools for detecting child sexual abuse material

#36

Post by continuations »

faustomorales wrote: post #8
nathanaldensr wrote: post #4

Earlier quoted context omitted.

Wouldn't a human have to go through thousands (or more) of illicit images and classify them in order to train the AI?

To build a classifier, yes, you are correct. But this isn’t a classifier to identify new content that has never been seen. This uses perceptual hashes to help organizations detect if known CSAM is being shared on their platform.

For an organization that uses this, wouldn't they need to have access to a source of constantly updated known CSAM? How is that going to work?

Re: Show HN: Perceptual hashing tools for detecting child sexual abuse material

#37

Post by ALittleLight »

faustomorales wrote: post #7
akersten wrote: post #3

Earlier quoted context omitted.

If I may, I'm curious about your thoughts on few things, in the context of use case #1 (abuse), not #2 (reduplication): * What are the challenges surrounding verification that your system functions properly, given that the test material is illicit? * Can you speak to the reliability of the system in a sensitivity/specificity kind of way? In other words, what are the false positive and false negative rates? * Are you…

Thanks for asking these important questions! * What are the challenges surrounding verification that your system functions properly, given that the test material is illicit? You're right that storing child sexual abuse material (CSAM) is illegal, unless you are the National Center for Missing and Exploited Children (NCMEC) or law enforcement. What is legal is to maintain a hash of known CSAM. NCMEC, Law Enforcement,…

It seems like an image classifier would work in this field. Are you aware of ML based image recognition and its utility here?

I'm imagining training data would be a hurdle, but surely you could give instructions or suggestions on how to train a model to people already authorized to work with the images.

Re: Show HN: Perceptual hashing tools for detecting child sexual abuse material

#38

Post by ALittleLight »

pbalau wrote: post #13

From what I've read, this is supposed to catch the bad thing after it happened, not to prevent it. Then you say that 1 in 1000 is an acceptable error rate. 1 in 7 billions is not an acceptable error rate!

If you catch people who have done bad things you prevent the bad things they are likely to do in the future.

1 in 1,000 does seem a bit high to me, but the prevalence of the bad images matters. You also have to think about the consequences of false positives versus admitting child pornography. If 0.1% of innocent meme pictures get removed from a site incorrectly - that's not ideal, but how much sexual exploitation of children do you need to stop to make that worth it?

Re: Show HN: Perceptual hashing tools for detecting child sexual abuse material

#39

Post by m0zg »

When I read about something like this, I can't help but wonder, how do you deal with error analysis and such? I personally would not be able to view such disturbing and revolting material. And if you don't do error analysis, how do you know this stuff actually works?
Post reply on HN