The added pressure and energy drain from working on your own machinations while working a full time job pales in comparison to the added pressure and energy drain from working against the clock. In my case it became a source of depression and a constant demotivator when I realized I was running into roadblocks. You cannot foresee all of the difficulties you will run into, you cannot rely on stories of other entrepreneurs doing so. They are half truths and tell only the triumphant side of it, ignoring the failures of the hundreds or thousands of others who's companies made it only to the paper stage then died with a whimper, much less being in the spotlight of the medium and techcrunch and online news articles and forums. You're hearing about certain entrepreneurs solely because they're outliers, not the norm - they're a good narrative.
Keep your job. Make the product. Make connections. Get Customers. Then make the company. Not the other way around. If you're the one in a million already has the connections and skills to make it happen then more power to you but statistically speaking, you probably don't. You should acquire those qualities with gusto but also with prudence.
Good luck.