I've put some serious thought into solving this problem. There are two main structural issues I know of:
1. Open source libraries tend to be complement goods. You're more willing to pay for a good physics engine if you already have a good rendering engine and vice versa. But a sad truth of complement goods is that they are a centralizing force - it's actually better for everyone if the physics engine maker and rendering engine maker join forces and offer a bundle discount. But the most common strategy seems to be for them to just merge into one company, and this is why you see giant conglomerate products like Unreal and Unity instead of buying each component from a different vendor.
2. Since open source software is a public good (non-rivalrous, non-excludable), the "free market" cannot really incentivize its production nearly as much as would be optimal. Let's say there are 1000 people who would each pay $10 for a feature to be added, and the maintainer would happily add it for $5000. If 90% of those people each paid $6 they would get what they want and the maintainer would be happy too, but each individual has an incentive to be part of that 10% that gets to keep their $6 and still gets the feature, so what happens is that almost no one ends up paying.
These problems can't be solved without slightly modifying open source, but they can be solved by maintaining the spirit of open source I think. What you need is to have some kind of foundation that takes money and gives it to "quasi-open-source" projects, and then only allows businesses to use those projects if they contribute a certain percentage of their revenue to the foundation. Of course, now the foundation needs to decide which open source projects to give the money too. It's an extremely tricky problem, but there's been a lot of interesting research by Glen Weyl on that exact subject and I'm confident it could be solved in a satisfactory way.
I think this proposal would create a virtuous cycle once it got off the ground. The more projects licensed "quasi-open-source", the larger the incentive to pay the foundation to use them. The more the foundation is paid, the more money these "quasi-open-source" projects get, and so more people will license their projects "quasi-open-source", increasing the incentive again, etc.
Of course, it would only be "quasi-open-source", and not truly open-source. But there's no reason the license couldn't be extremely in line with the spirit of open source. For example, it could say "if you're an individual or small company, you can use our code for any purpose for free. If you're a big company, you can use it in a way that complies with the AGPL or you can pay us, your choice".
I think employees would also encourage their employers to become paying members of such a foundation, if it lead to those employees being able to determine where some of the money goes. Everyone at my current company is a Rust developer and so we naturally like Rust, but Rust jobs aren't always easy to find. As employees, it could be in our best interest to subsidize the development of Rust open source projects, if that increased Rust's attractiveness to other companies.
If you're interested in this idea, my email is in my bio :D