Supercharging TensorFlow.js with SIMD and multi-threading
blog.tensorflow.org
Supercharging TensorFlow.js with SIMD and multi-threading
1–10 of 20 posts
Re: Supercharging TensorFlow.js with SIMD and multi-threading
#2If I read this right, this is much faster than the WebGL backend on the devices tested.
If the CPU is really faster than the GPU, that really demonstrates how inefficient the WebGL backend really is, compared to something like CUDA.
Re: Supercharging TensorFlow.js with SIMD and multi-threading
#3Looks alot like https://github.com/microsoft/onnxjs but onnx.js adds multithreading by web workers which will tske a long time to be available on wasm
Re: Supercharging TensorFlow.js with SIMD and multi-threading
#4As for traditional TensorFlow, the easiest way we found to improve performance (easily 2x) was to find/create builds tailored to our machines. Using Python, we had prebuilt wheels, which have (understandably) low feature requirements. If you find/build your own (e.g. if you have AVX-512), you can easily get pretty detect performance gains.
(Yes, there are unofficial wheels for various CPUs, but, not sure if that passes your security requirements.)
Re: Supercharging TensorFlow.js with SIMD and multi-threading
#5If I read this right, this is much faster than the WebGL backend on the devices tested. If the CPU is really faster than the GPU, that really demonstrates how inefficient the WebGL backend really is, compared to something like CUDA.
Note that these are light models that are designed to be run quickly on a cpu with batch size 1. It's not that uncommon to see multithreaded cpu code beat the gpu in that setting also for other backends.
Re: Supercharging TensorFlow.js with SIMD and multi-threading
#628ms on 2018 iPhone without threads or SIMD, 24ms on Chrome MBP 2019 with threads and no SIMD, 11ms with SIMD.
Re: Supercharging TensorFlow.js with SIMD and multi-threading
#7If I read this right, this is much faster than the WebGL backend on the devices tested. If the CPU is really faster than the GPU, that really demonstrates how inefficient the WebGL backend really is, compared to something like CUDA.
One of the advantages of using the CPU rather than GPU for inference (especially with batch size 1) is that it doesn't need data transfer from host to device, which is a notoriously slow, asynchronous process. This could also explain the difference in total run time, if measured correctly.
Re: Supercharging TensorFlow.js with SIMD and multi-threading
#8Couldn't tensorflow leverage webgl / webgpu?
Also it's really sad that there no webCL adoption yet
Re: Supercharging TensorFlow.js with SIMD and multi-threading
#928ms on 2018 iPhone without threads or SIMD, 24ms on Chrome MBP 2019 with threads and no SIMD, 11ms with SIMD.
What's the use-case for tensorflow on web/mobile web? I thought tensorflow was mostly for training models, and my assumption would be that this is mostly relevant for the server/workstation context, but maybe I'm missing something
Re: Supercharging TensorFlow.js with SIMD and multi-threading
#10Awesome work Marat.