Every result in physics hitherto has been some sort of mathematics: an equation or a constant (measured or otherwise established to some digits of precision). What is a particle? A collection of mathematical properties. So is a wave. If we extrapolate from the past to the future, we can expect more of the same: no "non-mathematical bottom" will be found. Nobody is even looking; the researchers expect all new results…
You're just describing the fact that we can make up descriptions of physics all the way down. If current mathematics is unable to describe physics, then new mathematics is invented [1]. Just because the descriptions work rather well does not mean that those descriptions are somehow more than mere descriptions. It just means that they are very good descriptions. You're making the same category error as mentioned in th…
Now suppose that this process stops. One day, mathematics describes physics. What does that mean? It means that some pencil-and-paper mathematics descriptions hint at an abstraction, and that abstraction is physics.
The mathematics which was used prior to that point was not the right one. That mathematics still corresponds to a universe, just not this one.
The idea is that every mathematical object is a universe (not to be confused with some representation of that object, like a definition in plain language, or a diagram, equation).
The world may be exactly the same category of thing as dodecahedron or pi. (Not in the category of pencil-and-paper description of such things; the category of those thing themselves.)