The moral basis comes from how the majority have agreed to structure society. A family isn't legally allowed to go to an unoccupied piece of land, start cutting down trees or making mud bricks into a shelter and stay there. We have divided up the land (a limited resource) into parcels of private ownership with government-enforced property rights.
With that comes a moral responsibility to make sure that everyone can obtain reasonable shelter. On the other side, for the person that owns the land, they provide access to what they own (as per the legal system that society has agreed upon -- private transfer of property, the banking system, etc). Not only do they provide access, they also provide the service of upkeep to the dwelling, upkeep of fixtures and security (locks, access control, etc). So it isn't like they are sitting back and collecting an income from the air that other people breath.