Before social media did anyone read every book ever published? Did anyone read all the rejected manuscripts to avoid the censorious hand of the publishing houses? Of course not, we accepted that someone (editors) were doing some first pass quality check and even then we pick what areas are interesting.
There's two related but distinct problems: the moderation problem and the village idiots problem[0]. Polarization _can_ come from moderation, but there's also a whole debate to be had about what is driving what. For example: Alex Jones' whole saga has been spun by some as "being punished for conservative beliefs", so yeah, I guess if he's a conservative then him being pushed off social might cause polarization. BUT I think it's important to note that 10 years ago if you said Jones was a conservative, almost _all_ conservatives would have said something "the interdimensional vampire guy? Don't lump us in with that crazy bastard". During the intervening years right wing leaders have increasingly signaled that Jones is one of theirs. That was a top down series of decisions more than social media's impact. In order to believe that "your team" is being punished you already had to believe that Jones was on your team. If the statement "Alex Jones is on my team and I'm on a mainstream political team" is true, then you're _already_ polarized. The moderation might make it worse but something severely fucked up has already happened.
The (potentially violent) extremism, though, is really about the idiots getting together and self reinforcing (for example incel groups periodically spinning out a mass shooter). Moderation isn't really going to impact the second problem since when they get booted from one platform they migrate to a less moderated one or spin up their own.
[0] Borrowing Peter Singer's framing from here: "Once, every village had an idiot. It took the internet to bring them all together."