Giving housing to the homeless means we change the homeless into neighbours. And who the hell wants a poor person as a neighbour.
Also, of course nobody wants to live with poor people. I don't buy this romantic image of poor people being fair citizens failed by the rest of the society. I moved into a poor neighborhood and immediately had my bike stolen, literally living the meme. Real estate prices are lower here exactly because it's a black immigrant neighborhood full of poor people.