I get a whole lot of that (even from my family: my sister told me "So, mom and I were talking in the car, and it's great that you're doing this startup, but honestly I don't think you'll succeed.") When I gave notice today, I had to listen to my boss go on about how I was too young to start a startup (I'm 26), how he spent about 15 years after getting his Ph.D learning about business and working in the industry, how my technical skills were too weak (nevermind that I wrote two of his products, and he's never seen me program in a language other than Java), and how if my idea was any good at all, I'd have been able to secure funding for it (nevermind that we're not interested in outside funding until we have some traction).
I've found that the best response is to sit there, listen carefully, take note of any valid points, and ask followup questions if you need more information on one. You may learn something: despite the overall negativity of the conversation, my boss had many points that I'm going to want to keep in mind as we move forwards.
Understand, there is a lot of self-justification going around when it comes to entrepreneurship. As long as rich people are the distant Bill Gateses and Warren Buffets, people can put them up on a pedestal or say "Oh, they got lucky." But if someone you've grown up with or someone who used to work for you gets rich, you have to ask yourself "Why them and not me? Are they just smarter than me?"
Many smart people will do just about anything to avoid admitting that others are smarter than them, so they instinctively say "Oh, he's just going to fail." And when you succeed, they'll say "Oh, he just got lucky." If you succeed again they'll start saying "The game is rigged!".
But if you stoop to their level and say "Oh, look how smart I am, of course I'm going to succeed," you're just engaging in self-justification yourself. And that's a dangerous mental attitude to get into, because it blinds you to details. The reason you're smart in the first place is because you pick up details that other people don't; you can easily become stupid by believing yourself smart. This comes from experience: I did precisely this in high school and college, and then found that when I actually tried to get something done, the results were much more disappointing than I would've liked.
(Therapy for myself: I think you're smarter than me, I think you will succeed, and I think that if you do succeed, it will be because of skill. But keep what I say in mind anyway. It may be useful.)