Many of the more reasonable criticisms of the memo say that it wasn't written well enough; it could've been more considerate, it should have used better language, or better presentation. In this particular link, Scott Alexander is used as an example of better writing, and he certainly is one of the best and most persuasive modern writers I've found. However, I can not imagine ever matching his talent and output, even…
If you want to build products exclusively for men, by all means, staff your teams with all men. But bringing a diversity of experience into all levels of a company makes that company's products and engagement experience better for everyone. This is especially true when you're talking about a consumer-focused product (like many of Google's products are).