I think what I've learned this week is that there's an economic failure mode that has societal harm. It goes something like this:
1. Some people want to share toxic harmful material with the world. Sharing that has real material hosting and serving costs that those individuals would rather not pay.
2. Platform hosts like AWS, Twitter, Facebook, etc. will host material for people. They pay for that by also showing users ads and the advertisers fund the company.
3. Advertisers don't want their brand associated with toxic content. But when a platform host has a sufficient diversity of good and bad content, the brand association is with the platform itself, which is neutral, and not the content it is shown next to, which may be toxic.
The end result is that advertisers end up inadvertently funding the dissemination of toxic content. Extremists are essentially free riding, sort of like riding on the back of one of those rolling billboard cars and tossing out pamphlets.
This happens because the host is essentially image laundering the toxic content. When a host isn't diverse enough to accomplish this, you can see that advertisers won't touch it. That's why Parler is toast. Advertisers do not want to be funding this stuff but the hosts they use are too much of a black box to give them much control.
So why do hosts end up accepting this toxic content? Is the relationship between host and toxic content authors parasitic or symbiotic? Cynics believe the hosts deliberately allow this because toxic content drives engagement and increases advertising revenue. A charitable perspective is that hosts would rather not be free riders for toxic content, but the problem is that separating the wolves and the sheep is too difficult (where "difficult" involves some combination of social, technological, and financial costs).
What I haven't seen is anyone suggest that it is an emergent property of financial incentives. The primary way these hosts compete in the marketplace is through network effects and economy of scale. Social media host success is dominated by network effects. Advertisers go where the eyeballs are, and people go where the other people are. I don't think there is any natural stable equilibrium that leads to a large number of small thriving social media companies.
Economies of scale exacerbate that. Users don't like viewing ads so the less ads a host shows, the more popular it becomes. This puts heavy competitive pressure on them to minimize costs which relies on all of the economies of scale that software and tech offers.
The end result, I think is a consolidation onto a few hosts which in turn host so much of everything that they end up with a neutral brand that advertisers can use even though there is a mixture of toxic content in there. I don't think anyone intended this to happen.
But the good news is that if we do create regulation to break up tech monopolies and push against the natural forces of network effects and economies of scale, we may also see a natural decline of toxic content free riders. Because as these hosts get smaller, they're less able to be used as image launderers.