There's a lot of pessimism in the comments. I believe this is one of the best uses for an AI that we've come up with. First, let's go into the details:
1. They only monitor and elevate to moderators content from posts and Facebook live.
2. Moderators can then take action, and reports from this system are prioritized over other non-pressing issues.
The actual life saving chances of this system are huge. Just from a pure safety area, it can help detect and alert someone in an authority area that something bad is happening when nobody else might. The bystander effect is a real problem, and I'd imagine that it might be even greater on social media. In many cases, posts that have suicidal tendencies are screaming for help from their friends. They're posting because they have no options left. In these cases, it's a safety net that can stop loss of life. Moderators make the final say, however, so it's not just AI diagnosis. It's AI warning that gets elevated to a human before anyone else. That's more eyes on the problem that can potentially help.
Are there other implications for AI? Yep. Advertising? Yep. Other types of tracking? Certainly.
But if we're going to use AI in those fields anyway, we may as well extend it to trying to save lives.