I was doing client work, and to be frank, got tired of working from home and dealing with low-grade clients all day. So instead of worrying about finding better clients, I took a day job to support myself and my family (wife and kids) while I worked on side projects with my co-founder.
To date, we've built a fun iOS game, Santa Strike, a crowdfunding plugin for WordPress, IgnitionDeck, and a few yet to be launched iOS apps, among other simple software utilities like Iconswitch.me and GameDesignTemplate.com
I drive to and from work an hour a day, which kills me, but we're very close to being profitable enough so I can quit my day job.
We're not zillionaires yet, but I do think it's possible to do what you're asking. However, it's very very difficult, especially when you have a spouse and/or kids.
I work on my stuff from 7-8 am, drive to work, try to fend off the thought of being an unproductive employee so I don't get fired (I'm actually the only developer they have, and I'm in a very good position because my predecessors set the bar so very low), but do spend some time at work handling side-project stuff, get home at 6:30, hang out, work more, go to bed, rinse, repeat.
My wife is stressed because my mind is elsewhere, but she understands. We spend time on the weekends going out and doing family stuff, but not as much as we'd like.
When we ship new versions of software, I field customer inquiries and complaints on the go, which isn't ideal, but is what it is.
In other words, I have very little personal time, it's very stressful for the entire family, and it's a lot of hard work over a long period of time. It's not for the faint of heart.