Even if it isn't the bottleneck, running the Ethernet over USB means the CPU has to get involved to do all of the Bus Mastering, which cuts into cycles it could be spending on preparing the packets.