Mainly, I was poking a bit of fun at the idea that "number" even has a rigorous definition. You can't really assert whether infinity is a number or not without 1) knowing what they mean by "number" or 2) knowing what they mean by "infinity." If you want "numbers" to be a field extension of the reals and you want it to contain an "infinity," then the hyperreal numbers might be appropriate [1]. If you are instead content with having an infinitesimal and having only a ring extension, then the dual numbers might be appropriate [2].
[1] https://en.wikipedia.org/wiki/Hyperreal_number
[2] https://en.wikipedia.org/wiki/Dual_number
But, anyway, if you can claim infinity is a number because someone thought of the real projective line, I can say polynomials are numbers. Square matrices, too --- I think of square matrices as being big numbers; rank measures how invertible a big number is.
A stranger consequence of my definition is that a continuous real-valued function on a space X is a "number." If X is a single point, then such a function is the same as a real number. If X is a discrete set of n points, then the set of functions is R^n. If X is compact (I think that's sufficient?) then the maximal ideals in the set of continuous functions is in correspondence with X itself. This suggests that for any ring, one may imagine there to be a space that it is the functions of, whose points are the maximal ideals of that ring. For the ring of complex one-variable polynomials, the points correspond to C itself (the maximal ideals are generated by (x-c) for varying constants c). So, yeah, polynomials are numbers now.