This may be an unpopular opinion but I still don't understand why diversity even matters here - shouldn't it be the merit of the ideas and the presentation skills of the speakers?
The trouble is, this is a very difficult position for the organizers to defend as it is easy to attack them with "So you claim that the other sex is not capable for skilled presentations, how dare you!" -comments. And likely this is something that resonates with enough people to create a storm in social media where you as the organizer end up in the middle. It is much easier to put up a more equal lineup of speakers…
I don't think it's particularly hard to defend yourself against that. You simply claim that for some reason women don't want to come to your conference so there wasn't enough submissions, and then promise to aim to do better. Maybe it's just me who feels that intent to fix is better than trying to cover up our problems. If i had any interest in this conference now I'd consider not going.