Frontend is really hard, writing re-usable HTML and CSS is almost as hard as writing re-usable backend modules. I get annoyed by backend engineers brushing off frontend, as it's a piece of cake for them. But when you actually ask them to write some, they struggle, because they have no idea how to make it reusable.
Frontend especially requires you to stay up-to-date, even more so than backend technologies. The ecosystem changes so much, that it can become an annoyance, but the key is to stick to what you think is best. For example, I still use Gulp for tasks, just because it works, and it's far easier to configure than Webpack or any other bundler.
But on the backend, I've found that there are far more interesting problems to solve. The design behind it is far more complex, especially when you start dealing with databases and message queues. But then you also need to start looking at how to deploy your backend, and how to scale it up, etc...
So when I made the move from frontend to backend, it wasn't easy. But it also wasn't particularly hard, if you know JS/TS, moving to a node backend is fairly straight forward. Once you have enough experience there, moving to Java or Python is just learning whatever framework you're using. What I found much more difficult, was moving back to frontend, after doing backend for a couple of months, I felt like many things had changed, and it was hard to switch my mindset back to frontend.