I wonder if Wolfram's 100-times-slower reference point was based on a very particular comparison. Suppose you benchmarked a numerical algorithm that dealt with a lot of matrix-vector operations. And suppose you compared, on the one hand, a C or Fortran implementation that used arrays and native machine finite-precision numbers, against a Lisp implementation that used lists and the sort of "exact" arithmetic described…
I could also see a matrix representation as lists of lists, and when you do anything you're always consing, and replacing elements of lists, and walking lists to get to elements. Compare that to a C-style 2D array of float or double, and Lisp looks horrible - plausibly 100x as bad, and worse as the size of the matrix increases. Of course, the problem with that is the Lisp data representation used, not Lisp itself.
Lispers aren't stupid, which should be distinguished from how easy it is make a simple Lisp. Making a performant one takes effort on the scale of making any similar language implication good and fast.