Most people don't need a conduct policy. Your average person isn't going to sign up for a website and start harassing other users, reposting Stormfront articles, and slinging racial slurs. On top of that, most people just click through the TOS without reading it. Do you even
read the gazillion EULAs you click through when you install software? Did you read Facebook's TOS when you signed up? I thought not. It won't affect them because they'll end up obeying the TOS even if they've never paid attention to what's in the TOS.
But if you don't have a TOS and enforce it, then you're eventually going to get the attention of the nastiest, most vile people on the Internet. And given that almost the whole world is on the Internet now, that's some of the nastiest, most vile people in the world. Once you get their attention, they're going to go out of their way to make your website their new home. If you were a Neo-Nazi, and you were used to being banned from every forum you found as soon as you opened your mouth, and then you found a forum that had no rules and allowed absolute free speech, you'd fall in love right away. So much that you'd go out of your way to advertise it to your Neo-Nazi friends and coördinate a mass migration. Once the nastiest, most vile people on the Internet migrate to your site en masse, they're going to start targeting people for harassment, they're going to use it as a platform to spread hate, and they're going to take the opportunity to recruit from the general population by posting carefully crafted propaganda.
And, yes, they'll drown out the decent people. First, because the vile ones are actively invading your site in droves, while the decent people just stumble on it from random links and maybe friends' recommendations. Second, because the vile people will drive the decent people out. You know the old adage about how a thimble of wine mixed into a barrel of sewage is still sewage, but a thimble of sewage mixed into a barrel of wine is also sewage? It's kind of like that. Once there are enough vile people to make their presence known, they will turn your site into sewage even if they're the minority.
tl;dr: Most people don't need their conduct regulated, but if you don't establish and enforce regulations, your community will be thoroughly overrun by the people who do need their conduct regulated.