I dream of quitting my job and living on some land. My wife refuses to "live in the middle of nowhere, where there's nothing to do". I guess I shall continue to work a job I hate in order to live in this expensive area.
Being out in the country where you can do whatever you want also means that your neighbors can do whatever they want. So some idiot setting off cherry bombs, or another one shooting skeet/target practice in his backyard, or heating a house with wood and smoking out the entire neighborhood when there's a temperature inversion is just as valid a use of property as your gardening. And when the various hunting seasons start, well, gunshots carry a pretty long distance at 6AM.
On a good day, I think the benefits outweigh the drawbacks. On a bad day, I just want to shoot everyone else, but that's me.