So let's analyze this for a minute, first at a general level of incentives that every software-building org has, then at the specific level of Spotify's founding culture and present incentives.
Generally: ideally we imagine UX & product roles as creating a strong experience rewarding the user on for their engagement, by user standards. In practice, in order advance their career, these roles need a narrative of wins they can sell to higher management. That usually means change one can take credit for, if possible paired with a metric management is invested in. Sometimes this metric is even connected with a rewarding experience for users. Sometimes. The incentive for change is there, though, regardless, at least as long as there's money in the dev budget for it.
Spotify Specifically: was founded under the premise that consumers could be drawn into a product that acts like a record collection buffet in the cloud, but only pay broadcast prices! (ie, free or flat rate). This premise was correct, but also relied on the idea that Spotify was entitled to pay artists broadcast-royalties while effect occupying the space where recording royalties used to live, cannibalizing the market for recordings. And so you get execs telling artists that it's "entitled" to want rates as high as a penny a listen for their music and Spotify wasn't built to solve the problem of artists getting paid it was built to solve the problem of piracy (artists not being able to get paid or set their prices is indistinguishable from the problems of piracy). [0] This tells us that either Spotify isn't getting enough revenue to pay a penny a stream, or it feels entitled to keep what it is collecting.
And here's what else we know about Spotify: their revenue per user is essentially fixed. Their model doesn't have much in the way to collect more from a user based off the users own values/actions. They can increase ads, but that's probably not a user driven win. What else can they do?
Well, they can do what the rest of the social media companies do with their feeds. Randomize appearance of content, mix in known engaging user rewards with other things that let Spotify sell user attention -- to a label or individual trying to promote music/audio in a given space, to third parties wanting to advertise or place another content.
That degrades user experience? Sure. You think the company that is founded down to the core on exploitation cares even a little bit about that? Especially now that they're such a huge name in music they outshine Apple in a lot of markets?
Eventually, I suppose, other people will be dissatisfied in the same ways you are and maybe in a snowballing number of ways caused by the kind of bad-incentives driven rot. But it's likely to be a long slow ride down the curve, with an exodus as fast and complete as FB.
[0] https://www.digitalmusicnews.com/2021/06/29/spotify-executiv...