I wouldn't have traded it for the world. However I was very lucky to grow up in an area with an exceptionally bad school system so there were quite a few other homeschooled families around. I learned math at my friend David's house from his dad who was a 3D visualization software engineer. I learned electronics from the neighborhood "elmer" who helped us all get our ham radio licenses. We did art and history at a bunch of places, and we did "field trips" with our friends to all sorts of places.
I've spent my whole life fighting the "ahh, an ill-mannered anti-social homeschooler eh?!" stereotype, but I have to admit that I've met my share of them too. This is my biggest concern with eventually homeschooling my own kids. Having people around, having a group, learning to get along with people, are all critical. Nevermind the fact that we learned all sorts of things that we weren't ourselves interested in because one of our friends was interested. Would I ever have gone on a field trip to learn to ride horses? No, but the neighbors wanted to, so we did! And they learned about hydroelectric dams when I wanted to go to those. I think I would've turned out a lot worse if I hadn't had that diversity of exposure.
I started community college at age 16 (paid for by the local high school) and was well prepared for it. (Except for chemistry. I still don't grok chemistry.) Neither of my parents ever went to college, and they've told me that they had the same concerns about being able to effectively teach. Kids are curious, and the resources to learn are out there. Don't get too hung up on following a curriculum and being an expert at everything yourself, get them with other kids and encourage curiosity. Learning how to learn is THE critical factor, and in my observation one of the things that public schools fail ridiculously at. (But I'm rather biased.)
Look into homeschooling groups in your area, or the newer related phenomena of "co-op schooling". (Essentially what I got.) Try and meet some of the parents in those group and see what it's like for their kids. If you've got any sort of support system around like that you'll do just fine!