We improved Tensorflow Serving performance by over 70%
1–10 of 18 posts
Re: We improved Tensorflow Serving performance by over 70%
#2FWIW, I believe that the current state of the art for batch-size 1, fp32 inference for ResNet-50 on Intel CPUs is AWS's work in https://arxiv.org/abs/1809.02697. After the low-hanging fruit outside of model execution are picked, this kind of work is probably quite relevant.
Re: We improved Tensorflow Serving performance by over 70%
#3FWIW, I believe that the current state of the art for batch-size 1, fp32 inference for ResNet-50 on Intel CPUs is AWS's work in https://arxiv.org/abs/1809.02697 . After the low-hanging fruit outside of model execution are picked, this kind of work is probably quite relevant.
Hey! Author here, thanks for linking the paper. The article was from an infrastructure perspective, but we're definitely diving deeper into graph execution optimizations after this:)
Re: We improved Tensorflow Serving performance by over 70%
#4https://www.microsoft.com/en-us/research/publication/deepcpu...
TensorFlow has some known inefficiencies.
Re: We improved Tensorflow Serving performance by over 70%
#5FWIW, I believe that the current state of the art for batch-size 1, fp32 inference for ResNet-50 on Intel CPUs is AWS's work in https://arxiv.org/abs/1809.02697 . After the low-hanging fruit outside of model execution are picked, this kind of work is probably quite relevant.
Hey! Author here, thanks for linking the paper. The article was from an infrastructure perspective, but we're definitely diving deeper into graph execution optimizations after this:)
Are there any particular optimizations you are looking into ?
Re: We improved Tensorflow Serving performance by over 70%
#6This is amazing.
Re: We improved Tensorflow Serving performance by over 70%
#7There is an optimized version of Tensorflow based on Clear Linux and MKLDNN -https://clearlinux.org/stacks , would be interest to see the performance difference between the natively compiled version and this .
Re: We improved Tensorflow Serving performance by over 70%
#8There is an optimized version of Tensorflow based on Clear Linux and MKLDNN - https://clearlinux.org/stacks , would be interest to see the performance difference between the natively compiled version and this .
Hey! That's super interesting - so far we went with Tensorflow's ubuntu based official Docker devel image, but a clearlinux base looks like it would definitely be worth looking into!
-masroor (author)
Re: We improved Tensorflow Serving performance by over 70%
#9The grpc.beta code elements are deprecated and may go away anytime. (gRPC 1.0.0 is also super old and unsupported)
Re: We improved Tensorflow Serving performance by over 70%
#10The grpc.beta code elements are deprecated and may go away anytime. (gRPC 1.0.0 is also super old and unsupported)
Good point - we're still in the process of migrating to >= 1.17. The gRPC connection and client stub should still translate (few semantic updates).
```
channel = grpc.insecure_channel('0.0.0.0:9000')
stub = PredictionServiceStub(channel)
```
-masroor (author)