"Discovering math", depending on what you mean, could be setting the bar pretty high. Memorizing formula is not what mathematics is about, but you're unlikely to get very far if you just "play with numbers". To learn mathematics and to think mathematically, you still need a structured environment: Lockhart's complaint is that modern (K-12) math teachers neither have that knowledge nor the ability to provide that structure.
If I were you, I'd go to where that knowledge does get taught: in undergraduate mathematics classes. That usually starts with some rigorous calculus (I second the recommendation for Spivak), real analysis (which might seem hopelessly unmotivated if you don't know/remember much calculus), linear algebra or abstract algebra, or discrete math. Depends on what you're interested.
Reading mathematics is hard. Google around for advice. Do lots of problems, which generally means finding proofs. This would very well fit the definition of "discovering mathematics".