I think its fair to say that not everyone shares that view.
This is probably the key to controversy in my mind. You could read it that way.
However, you can choose to interpret the sections about 'Hiring practices which can effectively lower the bar for “diversity” candidates by decreasing the false negative rate' and 'Alienating conservatives is both non-inclusive and generally bad business because conservatives tend to be higher in conscientiousness, which is required for much of the drudgery and maintenance work characteristic of a mature company.' as 'some people we're hiring don't cut it, we should hire more white men'.
That's not what it says. You don't have to interpret it that way.
...but clearly a lot of people did.
My advice? Stop defending it. You can talk about these issues without this memo.
Forget the memo. The memo was stupid and misguided. Releasing it internally at work was unreservedly stupid. ...but what's done is done.
Do you want to spend the next year talking about the memo, and the specifics of 'did it or did it not technically say such and such', or the actual issues he was trying to raise?