I'm 31. When I was 9, I wanted to copy Ultima, my dad had a copy of Visual Basic so I learned how to use that and made an RPG engine. When I was 11 or 12 I started getting frustrated with the performance of VB/QBasic (this is before we had Internet access, so reference material was sparse) but I had a copy of the programmer's guide to the IBM PC and PS/2 and managed to figure out how to use assembler to draw in mode 13h in BASIC. As I got on to be about 14-15, I started to try to use higher resolution modes and had real challenges with drivers, and moved on to using Pascal and C. I bought a Borland C++ package, it was hundreds of dollars.
My point to all of that is that, in the end, those experiences meant very little to me now. They were indicative of the kind of person I am, and they are pretty cool to look back on and brag about, but in the end, they don't really make me better at any specific programming task. If I were to have not had the same resources to try to do those things, I would probably not be significantly better or worse. I would have spent my time trying to take apart electronics or learning a language or something instead, developed my problem solving skills and semantic skills in other areas. Maybe a little bit less experience, but in the end I think I would not be significantly impacted.
The reason that people might fear that starting late holds them back is because they might not be the kind of person who, even with access to the tools and resources, would be interested in taking things apart and finding out how they worked to start with.
If you can say at this point that you're interested in stuff like parsers and data structures, then that's awesome. If you're interested in it and you want to learn more, you will learn more, and you will succeed. If on the other hand you are uninterested but think that programming is a smart career path, you'll probably do poorly.