One of the reasons is the difficulty of writing reusable interactive components with server-side rendering such as maps, rich text editors, autocompletes, hover previews, menus.
The components work with data. Let's say you need an auto-complete component - 500ms after the user starts typing it will query the server and get some data back. You can describe the returned data in partially rendered HTML too but the attribute-based descriptions are often quite limited and for ease of use from the JS side you're likely to end up with embedded JSON as a general data description language with unlimited nesting, which naturally leads you to a client-side templating language.
The components are also likely to re-use each other, giving rise to the need for a module system to track those dependencies and make sure they are all loaded when you go to a specific page based on the components being rendered. If you use partial rendering you would also need to ensure to track any components used from the partial html endpoints have their JS included in the page, capturing any of their occurences, making sure to attach the right event handlers to enable interactivity.
Sure, one could ostensibly write a set of components that use JS under the hood and add them to the back-end. They could also write a server-side module system that ensures that the right scripts are included in any page that uses them. They could do all that.
But then that would work only for a single backend ecosystem, such as e.g. Rails. There is no way this component library would be able to compete with a unified client-side ecosystem in terms of features, flexibility or battle-testedness. Additionally, if you want to try a different language on the backend, you won't be able to - unless you're willing to pay the overhead of a secondary service meant to only generate the HTML and manage the JS plugin system.
And all this is before we even consider alternative clients (e.g. mobile)