It doesn't matter at all where you went to college, or even if you went to college.
Sure, graduating from Harvard offers some short-term advantages, but if you're motivated you can take the lead.
Consider how much time in college is wasted studying minutia for a competitive exam that has no impact on learning.
Consider how much of college is often spent stressing about stupid deadlines that really don't matter or fawning over prestigious faculty who did something noteworthy 30 years ago and have coasted since.
My advice: Do something that really matters, and take the initiative to teach yourself as you go. You are (and should be) your own harshest critic. Make it your responsibility to find others to join you in your quest (if necessary). A college campus might be a good place for you to recruit.
But certainly don't hang your head!
note: most students at top phd programs didn't go to major name brand undergrad programs... they went to small (not well known) schools and accomplished meaningful research that would have been harder to accomplish in a cutthroat name brand atmosphere.