Plenty of discussion has been had around incentives and I agree with all that, but also there's another angle here: aside from dark patterns, I think the problems of terrible front-end experiences are most prevalent in the middle of the bell curve - the front-end products developed by rather mediocre skill levels and budgets. Which is usually a pejorative, but thinking of it as literally in the middle, not great but not terrible, a wide range of skill levels where people are more or less competent at their jobs. These are where I start to notice the worst effects of the median trying to adopt the technologies and practices that high skilled developers at good companies with huge budgets are able to utilize effectively, or outright make themselves, especially when there is so much churn that being "up to date" with technology is like a pipe dream to most people. These median teams inherit all the complexity, but due to budget or time pressures, etc. aren't able to overcome it effectively to produce a product with high quality, and users feel the sluggish experience and bloat that comes as a result. This pertains to more than UI of course but eventually manifests to users that way.
Something feels wrong. The median should be able to inherit the benefits of technology produced at the high end without such inheriting massive costs.