11–18 of 18 posts
What CPU is being used?
Assuming the benchmark is done with something like an EC2 C5 instance, the results in this post are quite slow. Somewhere around 14x slower than benchmarks from a year ago on EC2 C5 instances. [1]
[1] https://dawn.cs.stanford.edu/benchmark/ImageNet/inference.ht..., using the c5.2xlarge benchmark and assuming linear scaling
> containers are run on a 4 core, 15GB, Ubuntu 16.04 host machine What CPU is being used? Assuming the benchmark is done with something like an EC2 C5 instance, the results in this post are quite slow. Somewhere around 14x slower than benchmarks from a year ago on EC2 C5 instances. [1] [1] https://dawn.cs.stanford.edu/benchmark/ImageNet/inference.ht... , using the c5.2xlarge benchmark and assuming linear scaling
-masroor (author)
> containers are run on a 4 core, 15GB, Ubuntu 16.04 host machine What CPU is being used? Assuming the benchmark is done with something like an EC2 C5 instance, the results in this post are quite slow. Somewhere around 14x slower than benchmarks from a year ago on EC2 C5 instances. [1] [1] https://dawn.cs.stanford.edu/benchmark/ImageNet/inference.ht... , using the c5.2xlarge benchmark and assuming linear scaling
Hi bwasti, the host's CPU platform is Intel Broadwell. While the CPU architecture of our production hosts are the same, the resources allocated are much higher than 4 cores. This post details an overview of the relative improvements that can be made from a vanilla setup :) -masroor (author)
As an aside, I took into account the resource allocation in the parent comment. The c5.2xlarge has 8 cores, 8GB RAM [3] and does a single fp32 inference in ~17ms. If we chop that down to 4 cores and assume linear scaling we can fathom running ResNet-50 in ~35ms compared to the ~500ms achieved here. I'd recommend comparing to a known baseline rather than a "vanilla setup" to ensure you aren't missing any simple changes that may dramatically improve performance.
[1] https://github.com/IntelAI/models/blob/master/docs/general/t...
[2] https://www.intel.ai/improving-tensorflow-inference-performa...
Earlier quoted context omitted.
Hi bwasti, the host's CPU platform is Intel Broadwell. While the CPU architecture of our production hosts are the same, the resources allocated are much higher than 4 cores. This post details an overview of the relative improvements that can be made from a vanilla setup :) -masroor (author)
You may want to check out Intel's optimized version of TensorFlow Serving[1] for further improvements (on the order of 2x for ResNet-50[2]). As an aside, I took into account the resource allocation in the parent comment. The c5.2xlarge has 8 cores, 8GB RAM [3] and does a single fp32 inference in ~17ms. If we chop that down to 4 cores and assume linear scaling we can fathom running ResNet-50 in ~35ms compared to the ~…
-masroor(author)
[1] https://github.com/tensorflow/serving/tree/master/tensorflow...
Have a look here: https://github.com/IntelAI/OpenVINO-model-server/blob/master... You can replace tf-serving with OpenVINO to get even better performance and latency when running on CPU
Even basic image classifiers tend to be 100x faster on a GPU or TPU...
Have a look here: https://github.com/IntelAI/OpenVINO-model-server/blob/master... You can replace tf-serving with OpenVINO to get even better performance and latency when running on CPU
What useful models run at decent speed on a CPU these days? Even basic image classifiers tend to be 100x faster on a GPU or TPU...