I'd say what is polarized isn't necessarily "all the people", instead the perceived
discourse is polarized. And like in any really entrenched discourse, the one side thinks to know exactly what the other side wants and thinks, without ever talking to each other.
One of my best friends became a neonazi years ago. I don't live in that village anymore, but every 3 years I manage to meet him at some party or so. He has a extremely clear (and sadly utterly fantasized) image of my political goals and ideas. One of my goals is apparently to introduce muslim Sharia law nationwide, although I never ever mentioned anything about Sharia law to him.
It isreally hard to speak with him about anything at all, because he always seems to "know" what my opinion is, to the point of misrepresentation, while I am more or less just curious what he wants. What he says is usally a progression of: "I just want the foreigners to respect the rules!", "Deport the foreigners!" or "Shoot the foreigners!", depending on how many beers we are in. The whole thing is spiked with unsolicited explainations why liberal ideas "like mine" (although I am usually more on the realistic side) will never work and it sometimes feels he is telling this more to himself than to me.
This is rightwing Austria, his grandfather was a Nazi and this is where the discourse in the US is moving as well.