The market for new proprietary databases is both large and lucrative -- it is printing money right now. What
has changed is the structure of the market and the product economics. The classic marketing-driven retail database product model, the kind that puts billboards on the 101, stopped working as a financial model a decade ago -- development costs are too high for the marginal return. That has been replaced by models that produce much higher margins with much lower development cost. Unlike database companies of yore such as Oracle, they use little overt marketing because they don't sell through the public. It is a completely different value proposition for the customer too. And this market is growing.
Proprietary databases have found their niche with qualitatively higher performance and throughput -- often 10-100x -- than the best open source has to offer on the same hardware footprint. As data continues to grow exponentially and cloud costs increase, operational efficiency has a much bigger impact on the bottom line than licensing cost. This gap has only been increasing over time. For most open source databases this is a fundamental architectural issue, and you can't fix architecture, so they are defenseless against highly optimized competitors.
I was on a project a year or two ago where a particular ElasticSearch workload was replaced by a proprietary solution that demonstrably offered 100x higher throughput. It was apparently worth tens of millions of dollars to the customer for that improvement.
The reason big retail databases went away is that 90% of the development cost is in the "big retail" part and not the "database" part. At the same time, development costs for a state-of-the-art database engine have become extremely high due to wage costs (worse than FAANG), so the industry needed to find a business model that would allow those costs to be contained. It is still around but much more invisible. It would be nearly impossible to properly finance a database company as a startup, so a different model is used.