Faster Quantized Neural Network Inference with XNNPack
blog.tensorflow.org
Faster Quantized Neural Network Inference with XNNPack
1–10 of 13 posts
Re: Faster Quantized Neural Network Inference with XNNPack
#2Author here, happy to take your questions.
Re: Faster Quantized Neural Network Inference with XNNPack
#3Author here, happy to take your questions.
[deleted]
Re: Faster Quantized Neural Network Inference with XNNPack
#4Looking at posts from a couple of years back on HN/Reddit/SO about TF vs Pytorch, the only plus side of using TF was the ease of deployment, especially on the mobile side with Tensorflow Lite.
But I imagine that story is changing with the advent of Pytorch Mobile, ONNX, and that Pytorch itself supports XNNPack.
If anyone has any tips or insights as to ease of mobile deployment using TF vs using Pytorch, please share!
Re: Faster Quantized Neural Network Inference with XNNPack
#5Can it perform fixed point arithmetic with arbitrary number of bits?
Both training-aware and post training.
Re: Faster Quantized Neural Network Inference with XNNPack
#6Author here, happy to take your questions.
Is this a drop in solution that works with every existing tflite model?
Re: Faster Quantized Neural Network Inference with XNNPack
#7Author here, happy to take your questions.
Do the same optimizations apply to tensorflow/tensorflow serving?
Re: Faster Quantized Neural Network Inference with XNNPack
#8Can it perform fixed point arithmetic with arbitrary number of bits? Both training-aware and post training.
No
Re: Faster Quantized Neural Network Inference with XNNPack
#9Author here, happy to take your questions.
Do the same optimizations apply to tensorflow/tensorflow serving?
TensorFlow doesn't support quantized inference (it supports only mimicking quantization in floating-point for quantization-aware training), so it can't immediately benefit from these optimizations.
Re: Faster Quantized Neural Network Inference with XNNPack
#10Can it perform fixed point arithmetic with arbitrary number of bits? Both training-aware and post training.
It performs fixed-point arithmetic on 8-bit integers. You can mimick lower than 8-bit precision by using output_min/output_max parameters in XNNPACK operators, but keep in mind that:
1. This functionality is experimental and not exposed in TFLite. You'd need to call XNNPACK APIs directly from C/C++ code.
2. Computations would still be done on 8-bit numbers.