It asks why there isn't more evidence of aliens anywhere we've ever looked, not about why aliens haven't come to ask us for our wisdom on philosophical matters. It's not asking why they haven't contacted us, it's asking why we can't see them, or any evidence of them. In other words, it's about the statistical likelihood of seeing evidence that should be commonplace, given the age of the universe.
He brings up the point that we might be too primitive to even notice them. Fair enough, but you could just as easily say that it's "pretentious" to say elves don't exist, because we might just be too primitive and narrow-minded to see that there is evidence of them everywhere. We have to start somewhere, and we're doing our best.
The Fermi paradox may have a built in assumption that aliens affect their world through technology, and even that they are scientifically advanced and curious enough to send probes or ships out to explore. An attack on that point would be fair. It's possible that the answer to the paradox is that humans are extremely rare in having developed any level of technology, science, or curiosity. But, that seems more pretentious to me than the alternative.