It's partly a transition in the industry towards less risk taking. You see it in the movie industry as well.
Certain authors are the equivalent of blockbuster movie releases. Publishers pay them huge amounts and market the ever living daylights out of them. Sally Rooney is a recent example, or authors like Salman Rushdie or Zadie Smith, where their books releases are "events". Many publishers are moving away from the long tail model that used to let "midlist" authors still eke out a living or at least publish. The consolidation in the industry has also made it difficult for authors to shop their books around to different publishers with their own "brands".
The other "safe" bet for publishers has been genre books, like romance or crime, with clear templates and the ability to churn out books while relying on the authors name more than anything else, like Grisham. In romance, for example, there is a big emphasis on "family" series, where each book focuses on a different member of a family or group. An example is an author like Kleypas. The books are essentially just variations on a set of tropes, almost algorithmic, but the way they fit together and mutually reference one another keeps people invested.
In general though, very few authors make a living writing. The midlist era that thrived in the last 19th and 20th century is sadly going the way of the dodo.