>That's exactly what almost all knowledge is.
That's what almost all human assumption and belief system is, also ideology and religion, but knowledge is indeed something different, and not of a type that should rely on democratic consensus. It instead needs to be held up by material evidence that's always subject to retesting no matter how unpopular a new idea is. This is obvious.
The rest of what you say could just as easily be applied to the foolish social dogmas of nearly any past age in human history, dogmas that so often turned out to be wrong. A small number of reputable sources (for their time) upheld doctrines such as geocentrism, religious extremism, hatred for certain racial groups and numerous fervent beliefs in the right of certain people to dominate others. These are just a few examples.
A more material one would be the certainty among reputable sources that plate tectonics were nonsense, until of course they were shown not to be by what started as an argument by only a few people who were deemed very unreliable.
None of this is to give weight to every crackpot idea put forth, or claim that all opinions are equally valid until stated otherwise, but what makes the difference is evidence, not consensus.