Why benchmark using 32-bit FP on a V100? That means it’s not using tensor cores, which is a shame since they were built for this purpose. There’s no reason not to benchmark using FP16 here.
Not sure about the benchmark, but the code includes the option for mixed precision training via Apex/AMP.
The problem is that your conclusions aren’t independent of this choice. A different network might be far better in terms of accuracy/speed tradeoffs when evaluated at a lower precision. But there is no reason to use 32-but precision for inference, so this is just a big mistake.