If taking infinite density and an infinitesimal point seriously resulted in a clear and consistent ability to model it, then it would be as you say.
But that isn't what happens. Division by zero and other mathematical breakdowns occur, meaning that whatever actually happens is not in fact the same laws operating in an extreme situation. The laws don't actually work.
This is further backed up by the fact that we have two models with which to model the "singularity". Quantum mechanics and general relativity. They both break down, but in inconsistent ways. So clearly our equations don't work in that situation.
In addition, both from theory and experiment, there is strong evidence that space and time have a minimal length, the Planck unit of space and time. You can't get mass on a point if that understanding is true, because it will always involve unit distance connections.
Finally, uncooperative singularities like these have been found in scientific models many times, and the result has always been that the addition or adjustment of our models resolved mathematical mayhem. The mayhem just indicated the models were incomplete for the situation.
A toy example is Newton's force of gravity between two masses, F = Gm1m2/d^2. The force being a constant times the product of masses divided by the square of their distances.
This model implies that at a distance of zero the force is infinite. Yet that creates mathematical problems, modeling problems, and we never see it happen.
That mess is easily cleaned up with the realization that as the distance between the centers of a mass of m1 and m2 falls under the radius of one or both, r1 and r2, the forces of any mass of m1 and m2 outside distance d now cancel out. So as the position of two masses converge, the force of gravity actually goes to zero, not infinity. Newton's Law was still a good model for non-relativistic gravity forces, but needed to take account of more information to work in that particular case without blowing in a "singularity".
TLDR; where math blows up you can't just accept the model as operating in an extreme situation, because the model generates incoherent, inconclusive, undefined and inconsistent results. But the problem isn't magical, mysterious, or evidence of something unknowable. It is just evidence that our math and models don't yet capture everthing relevant to the breakdown case. It is a clue pointing toward something more for us to learn.