> Math has always existed.

I would say that laws of the Universe that math describes always existed but not math. Math is more of a language to describe something, and you can't say that the word "apple" always existed and was only discovered later. Apples existed long before the word got invented to describe it, though.