In Australia, investment residential property is seen as almost an inalienable right and a cultural norm.
I've lived in the UK and opinions there vary a lot. In London, it doesn't really have a negative connotation that I can recall. In rural parts, it tends to be viewed very negatively. I seem to remember reading a story about anger against "Londoners" driving up local property prices (with weekend homes) resulting in someone burning one down.
Weekend homes are a different beast of course but the anger comes from basically the same place: the perception that "outsiders" are making housing unaffordable for "locals".
This seems to be a very Luddite view. If it weren't for people letting out property rental homes wouldn't exist outside of, say, council-run rental buildings. I don't see that as a good thing.
Some places have tackled this problem with regulation eg making it difficult for foreigners to buy real estate or punitive capital gains taxes on short-term property sales like in Switzerland.
In NYC we have a different problem: rich people parking money in real estate leading to what are known as "ghost buildings". This is mainly a Manhattan issue.
Historically NYC had rent control that ended in the early 70s (meaning no new rent control leases were issued, existing ones remained while the tenants remained). Reforms were enacted to allow tenants to buy their apartments leading to the "coop", which accounts for the vast majority of Manhattan apartments.
A coop is an interesting beast. Technically you own shares in the coop. The coop owns the building. Your shares entitle you to reside in a given apartment. You must comply with the coop's rules. Coops are typically for primary residences and, to a lesser extent, pied-a-terres.
You can sublet coops you own in some cases but it tends to be pretty restrictive (eg only for 2 years out of every 5 and you have to own it for at least 2 years).
The effect of this is that coops tend to have very high occupancy rates (in real terms) and relatively low turnover.
Most new builds are condos, where you own your property outright and are a member of an HOA (Home Owners Association). Buying a coop will involve getting approval by the board, which can place wildly varying limits on mortgage amounts, income to debt ratios, requiring personal and professional references and so on. Condos however require only finance approval (the building has a right of first refusal).
So a lot of condos are owned by wealthy people who might visit for a few days a year if at all. I don't think this ghost building phenomenon is healthy for a community and is certainly a problem if all construction falls into that category.
Buy-to-let isn't the problem here. Unchecked buy-to-let might well be. Banning it is throwing the baby out with the bathwater. A more nuanced scheme might give tax abatements for owner-occupied properties in the coop model. Or put quotes on investment property in a given area. You can even create markets for such things eg NYC has a market for "air rights" above buildings to allow really tall buildings.
As far as "crappy" rental property goes, that's pretty much rental property all over.
