ONNX Runtime and CoreML May Silently Convert Your Model to FP16
ym2132.github.io
ONNX Runtime and CoreML May Silently Convert Your Model to FP16
1–10 of 19 posts
Re: ONNX Runtime and CoreML May Silently Convert Your Model to FP16
#2[flagged]
Re: ONNX Runtime and CoreML May Silently Convert Your Model to FP16
#3[flagged]
Eh, those “ai researchers” are too busy rolling around in mounds of freshly minted Benjamins to care about “quality software”
Re: ONNX Runtime and CoreML May Silently Convert Your Model to FP16
#4[flagged]
Not until it gets tensor parallelism.
Re: ONNX Runtime and CoreML May Silently Convert Your Model to FP16
#5[flagged]
While this is a bit too harsh - and the solution is naive at best - the problem is real.
The idea of bitwise reproducibility for floating point computations is completely laughable in any part of the DL landscape. Meanwhile in just about every other area that uses fp computation it's been the defacto standard for decades.
From NVidia not guaranteeing bitwise reproducibility even on the same GPU: https://docs.nvidia.com/deeplearning/cudnn/backend/v9.17.0/d...
To frameworks somehow being even worse. Where the best you can do is order the frameworks in terms of how bad they are - with tensorflow being far down at the bottom and jax being (currently) at the top - and try to use the best one.
This is a huge issue to anyone serious about developing novel models and I see no one talking about it, let alone trying to solve it.
Re: ONNX Runtime and CoreML May Silently Convert Your Model to FP16
#6[flagged]
While this is a bit too harsh - and the solution is naive at best - the problem is real. The idea of bitwise reproducibility for floating point computations is completely laughable in any part of the DL landscape. Meanwhile in just about every other area that uses fp computation it's been the defacto standard for decades. From NVidia not guaranteeing bitwise reproducibility even on the same GPU: https://docs.nvidia.c…
> Meanwhile in just about every other area that uses fp computation it's been the defacto standard for decades.
Not that strongly for more parallel things, quite similar to the situation with atomics on cuDNN. cuBLAS for example has a similar issue with multi-stream handling, though this can be overcome with a proper workspace allocation: https://docs.nvidia.com/cuda/cublas/index.html?highlight=Rep....
Still better than cuDNN where some operations just don't have a reproducible version though. The other fields are at least trying. DL doesn't seem to be.
On that note Intel added reproducible BLAS to oneMKL on CPU and GPU last year. https://www.intel.com/content/www/us/en/developer/archive/tr...
Re: ONNX Runtime and CoreML May Silently Convert Your Model to FP16
#7if you double click the coreml file in a mac and open xcode there is a profiler you can run. the profiler will show you the operations it's using and what the bit depth is.
Re: ONNX Runtime and CoreML May Silently Convert Your Model to FP16
#8[flagged]
While this is a bit too harsh - and the solution is naive at best - the problem is real. The idea of bitwise reproducibility for floating point computations is completely laughable in any part of the DL landscape. Meanwhile in just about every other area that uses fp computation it's been the defacto standard for decades. From NVidia not guaranteeing bitwise reproducibility even on the same GPU: https://docs.nvidia.c…
> The idea of bitwise reproducibility for floating point computations is completely laughable in any part of the DL landscape. Meanwhile in just about every other area that uses fp computation it's been the defacto standard for decades.
It is quite annoying when you do parallelization, and idk if that many people cared about bitwise reproducibility, especially when it requires compromising a bit of performance.
Re: ONNX Runtime and CoreML May Silently Convert Your Model to FP16
#9On the coreml side this is likely because the neural engine supports fp16 and offloading some/all layers to ANE significantly increases inference time and power usage when running models. You can inspect in the Xcode profiler to see what is running on each part of the device at what precision.
Re: ONNX Runtime and CoreML May Silently Convert Your Model to FP16
#10This was an interesting read, thanks for sharing. I've recently been building something that uses Parakeet v2/v3 models, I'm using the parakeet-rs package (https://github.com/altunenes/parakeet-rs) which has had a few issues running models with CoreML (unrelated to the linked post), e.g. https://github.com/microsoft/onnxruntime/issues/26355