Go further. Get rid of all zoning that’s unrelated to pollution of some kind or strictly a safety issue. Why is the government involved in whether I can run a business from my home? Why is the government responsible for controlling where businesses might be located instead of the free market controlling that aspect more dynamically?
Well my neighbor works in construction and would park several of his giant dump trucks in the street parking and the crew would gather outside every morning making a ruckus while leaving for the day. It was terrible and I was glad the code department put a stop to that and made every person in that neighborhood happy. This reasoning is exactly why I stopped being a libertarian, the free market has no recourse mechani…
Nuisance law is a recourse mechanism in the case you point out, and is included in any reasonable notion of "free market". (In the more radical notions of libertarianism, you and the other guy would simply submit the dispute to a mutually-trusted outside arbitrator, with a voluntary agreement to abide by the arbitrator's ruling.)