One thing that I've learned over time is that if I can't understand why people are taking some particularly obtuse or circuitous route, then that's probably because I don't yet understand what they're doing properly. I never really understood why people would use terminal-based text editors until I tried it out for myself, and while I still have my doubts, I understand the tradeoffs much better now. If frontend development feels as insane to you as the analogy you're describing, and yet it's one of the largest sectors of the software industry, then it seems likely that you're missing something, rather than everyone else going mad.
As you say, it has always been possible to build things where the browser-based client is separated from the backend via an explicit API. Consider an application that shows you a list of flights coming into an airport. You want it to automatically update whenever new flights come in, so you need some polling to fetch that data, and some Javascript to clone elements, fill in the details of the new flights, and insert them into the DOM. But you also want sort functionality, so you need to be able to rearrange all the elements as well. And you want to filter, so you need to be able to store all the flight information internally as state, even if those flights aren't currently being shown. And your can click on each flight to expand it and view more details about that flight, so while you're rearranging and rewriting all your flights, you also need to be keeping track of which ones are currently open or not.
This was the exact application where I first realised that all the jQuery-based DOM manipulation techniques I'd learned were not going to be enough, and that there was a lot of value in having a tool that manages keeping the DOM in sync with the application state, allowing you to focus on keeping track of that state.
There are other approaches, but in my experience, they mostly come down to building your own framework, one way or another. They don't have to be as complicated as React, or as heavyweight as Angular, but you need some system in place to help you sync your state to the DOM.
I'm not establishing here that all web development needs to take this approach, just that there are a lot of situations where it is very useful. I've ended up building and maintaining variants in the above filter/sort/details/live updates system three or four times in different contexts and with different parameters, to say nothing of other, similarly complex web apps where the system works have been unmanageable without some sort of frontend framework.
Once this starts, though, you end up with frontend developers who specialise in this kind of development, and I suspect this results in a kind of siloisation, splitting web development into frontend teams and backend teams, who can each specialise further. Backend developers no longer need to worry about that occasional time where they need to tweak some CSS or JS, and frontend developers don't need to understand databases. This in turn develops into frontend developers using their tools in a wider variety of situations, because it turns out they make a lot of web development a lot easier, not just complex state management. It's a bit like how people use git to track their dotfiles. Even though git is designed to coordinate between multiple people syncing complex branches and forks together, it turns out it also works well as a simple backup and rollback mechanism.
The result is that tools like React are probably somewhat overused, but that allows developers to specialise into broad fields like "frontend" or "backend", while also remaining very general within those fields (because most work that is frontend will use similar sorts of tools and layouts, just like how MVC will apply to most backend development, regardless of language or framework). And this also isn't universal - there are still plenty of smaller, more boutique design operations that will build websites in the more traditional way.