It think this is silly on multiple accounts. I'll claim that there's not real thing such as a "language brain" or "math brain." I'll also claim that most people don't know what math is, and that their evidence supports a "math brain". Math isn't about calculations/computations, it is about patterns. You get to algebra and think "what are these letters doing in my math" but once you get further you think "what are the…
By the very expansive definition that you advocate for, what exactly do we do that isn't math? One could argue that all we do is turn thoughts, senses, and memories into further thoughts and appropriate actions, which is applying a pattern, which is math. But at that point the definition is too broad to be helpful for anything but playing word games.
But yes, math is very broad. But it doesn't cover everything. At least not at this time.