Add in the cost of the human waste disposal that you make unnecessary (ah, I guess that's the "nitrogen pollution" bit). But I understand that phosphates are at least as serious a problem; and I believe nitrates are much less persistent than phosphates.
If I remember a fragment of chemistry from 50 years ago, it's something like "all inorganic nitrate salts are soluble in water at STC". Ammonia's also water-soluble (and volatile); rivers and groundwater can recover from nitrogen pollution more quickly than from phosphate pollution.
One other thing: phosphate fertilisers basically have to be dug out of the ground, and as reserves begin to run low, prices are increasing. But human poo is very high in phosphates.
I mean, even if there's a yick factor, let's face it: every glass of water we drink, whether from the tap or the mineral spring, has probably passed through the kidneys of dozens of animals, including fish, mammals and insects, and certainly hundreds of dinosaurs and millions of bateria. Why should we be concerned if the phosphate salt we use was once a part of human poo, which probably contained nasties at some point? And nobody's proposing that humans should chow-down on phosphate salts extracted from human poo; stick to bananas, and put the fertiliser on the banana tree.