>...is to subject all freedom of sentiment to the prejudices of one man, and make him the arbitrary and infallible judge of all controverted points in learning, religion and government
This doesn't seem right. Firstly, we generally operate under the principle that laws are created by the populace, not "one man", the censor. That seems to contradict the idea that a control on freedom of speech would be up to the whims of one person, if the censor is restricted in his activities by the voting populace and the lawmakers they elect. Secondly, this type of restriction has precedent in several other important areas of life, such as food and drugs. While many people disagree with the FDA's decisions, very few people argue for its abolition in its role as a 'censor' of which products can be sold and in what quantities on the American market.
Of course, the usual retort would be "I trust the state to regulate food and drugs more than I trust it to regulate speech", but to convince someone else of that point, you'd have to also provide evidence and reasoning that our hypothetical censor cannot be trusted to censor according to the law that the citizens vote for via their representatives. Another retort would be "Free speech is more important than food and drug availability", but that statement implicitly admits there's an issue with any food regulation, it's just not as important as speech, which leads to the question of why so few free speech absolutists care about abolishing the FDA.
I haven't really seen any convincing argument for a slippery slope when it comes to regulation of speech, nor any reasoned argument on why there ought to be an absolute constitutional protection on speech. That said, I'm not personally in love with the idea of a censor, as I've seen that censors in other countries have practiced unjustified censorship - but that seems like a problem with the democratic process and oversight rather than the idea of a censor.