Publishers have a good idea who will succeed. Being able to estimate that is part of the job, and they've become far less diverse and adventurous over time because they've become far more corporate and profit=focused.
Most published books either have a track record as self-published break outs, and/or they're trend chasers, and/or - as the article noted, but didn't say enough about - they have authors who fit a known demographic with known tastes and can be marketed on the strength of the author's story.
The real reason so many books are published is because it's a volume game. Most books are actually profitable - barely - so they need to keep churning them out to make adequate aggregate profits.
It's true that most of the money is made by a few tens of authors, but that doesn't mean no money is coming from the rest.
The tiny number of high prestige titles are cultural loss leaders which give publishers a cover story that they're really about Serious Literature and aren't just content mills.
This used to be a credible excuse when Serious Authors were still a thing, but it's looking more and more threadbare these days.
Serious Authors used to be household names, but hardly anyone can name a Recent Serious Author now. Critics still criticise, competitions still award prizes, and review editors still review, but most of the money and most of the interest is elsewhere.