The blog post appears to get confused and devotes its entire second half to pitching Keenable itself. If the idea is to build credibility for the new benchmark, this maybe was not the best choice. Search engines with their own indexes can retrieve documents no one else has. That means your agents will be able to answer questions previously unanswerable. Besides the clear AI smell, this nonsensical claim also plainly…
> Besides the clear AI smell, this nonsensical claim also plainly contradicts the methodology's key evaluation claim that the quality of an engine's results should be measured against how much it overlaps with the reranked aggregate of the other engines. The benchmark thus seemingly values an engine's ability to "answer unanswerable questions" at zero.
The engine itself is part of the reranked aggregate so if it finds something useful and everybody else does not it gets full credit.