As an outsider, I've always understood it like this. The conservatives and right leaning people value individual rights and freedom. For example, they are against something like mandatory healthcare, because they might be perfectly healthy yet would have to pay to carry those who are not. Democrats and left leaning people on the other hand value a more general sense of freedom and health. They want to carry the weakest people because they believe that they could at some point be in the same situation, where they would have to rely on others. (Bare with me, English isn't my first language)
So, when you say "Can’t we all try to find consensus and compromise?", in my opinion that already goes against what conservatives and right leaning people value. An individual having to compromise for "the greater good" goes against their core values.
Now, I do think that life is more complex than simply choosing one side. Because life isn't necessarily copatible with "human rationality". Sometimes there are situations where it is more rational to be hypocritical or sometimes it is more rational to let conflicting opinions exist.
I think it comes down to what humans learn at a young age and in school. If we were able to focus more on ethics and helping people find a purpose to live at a young age already, it would make a big difference.