This article reflects my wife and my experience: just a month ago we left San Diego for a small town north of Salt Lake City. We make over six-figures and San Diego held only poor economic prospects for us. We'd have a very difficult time buying a decent home (2000 sq ft, good neighborhood, reasonable commute) for a price which wouldn't jeopardize other essentials like retirement savings, kids college savings, saving…
You don't come across as negatively "privileged", but perhaps if you can afford to send your kids to college, and plan for retirement, and have a rainy day fund to boot, you might consider yourself blessed compared to the situation many people find themselves in. I'm well enough off, but I'd give an arm to be in your position. :)