UX is a by-product of the mess we have made of web development.
We made web development so overly complicated that vast teams are needed for the simple task of showing stuff on a page.
Part of the holdup has been CSS. To build out a responsive layout used to be phenomenally hard. Tim Berners Lee didn't think layout was needed, originally the web was to just link documents that people would open in other programs, so the complexity would have been in those other programs.
Need a spreadsheet? Then you would get the link and open it in your spreadsheet program, not an online Google Docs equivalent with its own special interface.
With the difficulty and lack of tools in CSS it meant that web pages had to be hacks. Along the way cruft such as frameworks came along to make it that bit more possible, but imposing more stuff to learn along the way. Therefore it meant that teams had to get ever more specialist, you could not just have 'webmaster' doing it all.
Things got increasingly siloed. Then this agile nonsense came along to slice and dice projects. This made frontend dev a painting by numbers exercise with designs handed down from on high. Those designs would be done by a designer who by definition did not know HTML, they would be cribbing from other stuff and not acknowledging their sources, meaning that frontend dev was an exercise in reverse engineering whatever was in the PDFs and imagining the way it was supposed to work.
Bringing on a UX person with the bullshit language about personas and other nonsense that went with the job took the dev team even further away from understanding the customer, the task overly specialised.
Along the way we moved to meaningless HTML, back to that early web stuff we were supposed to get away from. Instead of FONT tags in the markup we ended up with these silly divs and non-semantic class names on every element put there for layout hacks.
The thing is that anyone at the coalface of development is assumed to be useless at design, whereas the kiddo out of art school that can't code is assumed to be a genius at it.
If you have done 300 test orders of a checkout then you find the pain points of the process and can fix them (or ignore them). If you just do drawings in Photoshop of how it is supposed to work then you ain't gonna be having these insights.
So rather than trust the dev team and let them make decisions the designs are cast in stone and these UX experts (who can't code) call the shots.
We have developed these huge bloated teams and denied entry to people who want to code with Notepad and FTP. I don't use Notepad and FTP myself, but I don't think that people starting there need to be excluded from the web which should be for everyone.
Luckily a lot has changed.
We now know what works with UX, so there is no point in having a UI bod putting the menu in the bottom right because they need to 'design' something. Or changing the search icon to a pair of spectacles because they are an artist. Those things are now standardised, we have got that.
Equally devs who deliver the deliverables rather than mockups know these things.
Also changed is that browsers are standards compliant. No need to produce static mockups when you can do a mockup in HTML.
Also changed is CSS grid. There is now a layout engine in every browser that does all the things the hacks were needed for. This innovation means that layout is no longer a major job and real content without the useless div and class bloat can be directly styled in a quarter of the time it used to take with clumsy frameworks.
Another change is accessibility. There are laws coming in to make that a thing. So putting these pieces of the puzzle together I hope there soon comes a time when web dev teams get usurped by much smaller and more nimble teams that don't over-complicate the gig and keep it simple. After all web pages don't need to be that hard.