Even if you're certain you'd win a lawsuit, you'd still have to be able to afford that lawsuit. As we saw in the Google vs Amazon case of the Java API, such lawsuits can be long and complex, and most RPG publishers are tiny. WotC is probably bigger than the entire rest of the industry put together.
Even if technically game mechanics are fair use, where exactly to draw that line has never been tested in court, and in the past, well before the OGL, RPG developers publishing something that was intended to be compatible with someone else's game, have been sued over that. Including WotC itself in their early days.
So just that already gave WotC an incentive to want a more open license. And in the late 1990s, before they released the OGL, the RPG industry was shrinking, with each company on its own island, and no exchange of material between different systems. WotC reasoned that the value of an RPG comes primarily from how easy it is to find players to play the game with, and if all RPGs are built on the same core mechanics, they have a common player base and players can much more easily pick up a new game. That's why they created the OGL: to connect all the islands, create a common player base for everybody, make all RPGs more valuable as a result, and thereby make the entire industry bigger. A bigger pie, of which WotC got the biggest piece.
On top of that, they hoped that third party contributions would also improve D&D itself. Inspired by ESR's the Cathedral and the Bazaar.
But the real thing that convinced other publishers to contribute to this common OGL system, much more than any of the legalities, is the trust that WotC built. The OGL was a solemn promise that they would never sue anyone who stuck to the rules in the OGL. Instead of the vague complexities and grey areas of copyright law, there were now clear lines drawn about what you could and couldn't do. And that is what convinced other RPG publishers to go along with this. To trust WotC. And everybody benefited from it.
And now WotC destroyed that trust, and may never be able to repair that. Paizo is trying to rebuild a new similar agreement and the accompanying trust from the ruins of WotC's sudden idiocy.
WotC seems to be seeing the RPG industry as a zero-sum game, where any money made by someone else, is money not made by them. That's not how it works. WotC benefited enormously from the OGL. The biggest editions of D&D are those that used the OGL (3.x and 5), whereas 4 was significantly less popular, and at some point even outsold by Paizo's Pathfinder. RPGs have always been a community effort. 20 years ago, WotC understood that. Modern Hasbro leadership only thinks in terms of products and consumers, and that attitude is going to hurt D&D.