I think it's possible to 'grow' in something you don't care about or are not necessarily your core strengths.
Like I never had much interest in DevOps, but previous jobs have put me in a situation where I had to learn it and although I sucked at it at first, I got better at it, and since it seems to have become almost an unescapable part of a full-stack web developer's job nowadays.
Also I haven't cared much about the domains of several companies I've been hired for or clients I've been put to work for as a consultant (like I actively tried to avoid the insurance and finance industry, and ended up doing work for both). Hasn't stopped me from learning that anyway, and I can follow along (mostly) with our client when they talk about wealth management concepts now, when I was totally lost in those conversations when I first started.
And at a previous job I knew nothing about developing phone systems, but did a little work here and there, and eventually became the department's 'phone systems expert' as I became the only person left in the department that knew enough about those systems as people transferred or left. I handled the role just fine until I left for another job. I don't do anything with phone systems now, but I do have a better understanding of how networks work because of that job, which has been useful.
It is an opportunity cost, though. I have lost some of my other skills because of time spent in these roles. Like I think I'd have trouble going back into video game development after spending so much time out of it, at least at the level of seniority that I would kind of need in order to get sort of in the ballpark of the salary I'm currently making.
I'm still making games in my spare time, but I don't have a ton of free time anymore, so progress has been slow and the games I've been working on have been tiny in comparison.