Mathematics gets way more attention than it deserves in computing -- particularly with today's sophisticated libraries/frameworks. Just because a topic can be challenging doesn't automatically make it worthwhile. This is a trap that many people fall into.
I think the more common trap, into which you appear to have fallen, is thinking that computer science and software engineering are the same thing.
A corollary of this is that it's impossible to speed up "business logic" at all. You have to drop from Java (typically) directly into assembly language.