Numbers are useful abstractions with no empirical evidence to back them up. That's my point. Numbers are a way of interacting with the physical world and reproduce certain results, but numbers themselves are abstract and have no physical properties.
As for empiricism, I certainly did not throw it out. It's useful, but not for everything. Say I have three knives in my kitchen drawer: a butter knife, a steak knife, and a chef's knife. Empirically speaking, which knife is the best knife? You can tell me all the physical properties of the knife, and the metallurgy of the knife and the shape and weight of each knife, but which knife is the BEST knife, from an empirical point of view? Empiricism has no answer UNTIL you, a person, say "I want to dice this onion" and empiricism can then arrange the knives in a hierarchical order according to which one will work best. But science has nothing to say about "best" or "worst" until an agent in the world has a purpose for which they want to use something. Then empiricism is a wonderful tool.
Another, shorter example: empiricism can help you build a rocket, but has nothing to say about whether you should aim it at the moon, or at a neighboring country.
And one more, just for fun: the empiricism that was used to detect and classify climate change is the same empiricism that was used to make the internal combustion engine. Empiricism is blind as to intent or motive. Empiricism is a toolset, but has no tools for determining a value hierarchy within the bounds of empiricism itself. Empiricism can only go to work once a person has given it a purpose or goal to work towards.