Go further. Get rid of all zoning that’s unrelated to pollution of some kind or strictly a safety issue. Why is the government involved in whether I can run a business from my home? Why is the government responsible for controlling where businesses might be located instead of the free market controlling that aspect more dynamically?
This reasoning is exactly why I stopped being a libertarian, the free market has no recourse mechanisms for actors making peoples lives worse.