Have you found a problem/pain point (at your current job perhaps) that you want to solve and turn into a startup? That's the first thing to consider. The two companies I've started began with "we need to solve this problem" and (in one way or another) then evolved into "hey, this could be a business" ... if you just start with "I want to do a startup", I think there's a higher risk of you making something people don't want/need.
That being said, I don't think it's ever too late.
But being married with kids makes the decision a little harder. Doing a startup on the side (i.e. keeping your job for now) means you sacrifice time with your family. Taking the leap and quitting your job (at some point) may mean a lifestyle change for everyone if you're only ramen profitable or living off savings. Those are decisions you need to make with your spouse. There were some months early on I really put some stress on my wife (financially) ... that's not a discussion where you sugar-coat things or pull a "if only 1% ...".
I'm in my mid thirties, married, and have 4 kids (one is already 15!) so I occasionally feel like an "old founder" when I read about the college kids on HN or TechCrunch. But we do real well now. I think its been 8 years since I had a "real job" ... no regrets.