I actually think the hardest part about ML is the lingo. It's very alienating that even simple concepts seem to have their own lingo. A lot of the ideas are just what you as a developer might do intuitively if you had to implement it. But the language tends to be a bit mathy and obscure. So, when you try to read something without understanding the lingo, it seems impenetrable. But once you know things like "quantization is basically rounding" . . . it becomes easier.
Since ML comes from statistics, math, programming, but also other scientific fields, it can even have many terms for essentially the same thing.
For me, as a developer, it was actually easiest to just read some tutorials like the docs for scikit learn and then just start digging through the code of a bunch of libraries. How people name the classes tells you what they think things should be called. But the code tells you what it actually does. I just bounced back and forth between code, tutorials/blogs and books. After a few months, I can actually have a reasonable conversation with our ML people in the language they use and everything else I look at seems easier because I understand most of the terms.
I think asking how to learn ML is a lot like asking how to learn German. It might feel like you need to start with the grammar rules. But I think immersion is the best way. Get the vocabulary, then come back to the rules. I also find that having a burning question in my mind helps me with immersion. So, if you can find a project that drives you, maybe that will help.
So starting with the math fundamentals as a developer seems like an easy way to burn yourself out. But everyone does learn differently. If not, there wouldn't be so many ML algorithms, right? Right?