Bfloat16 support coming to Apple's Metal and PyTorch [video]
developer.apple.com
Bfloat16 support coming to Apple's Metal and PyTorch [video]
1–10 of 55 posts
Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]
#2Somehow missed this from WWDC23, but it looks like Sonoma will add support for bfloat16 with Metal, and there's an active PR to add support with the PyTorch MPS back-end (PR #99272). Since M2 added bfloat16 support at the hardware level, I'm assuming this will only be supported on M2 Macs.
That maxed out Mac Studio M2 w/ 192GB of memory now looks more appealing...
Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]
#3I'm still confused by the proliferation of bf16. Although it certainly doesn't hurt compared to fp16, in my testing even with A100 GPUs optimized for it, both training speed and inference quality are the same between bf16 and fp16.
Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]
#4I'm still confused by the proliferation of bf16. Although it certainly doesn't hurt compared to fp16, in my testing even with A100 GPUs optimized for it, both training speed and inference quality are the same between bf16 and fp16.
bf16 is generally easier to train neural network than fp16 on due to no need for scaling. And most model training and inference performs the same with fp32 and bf16.
Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]
#5I'm still confused by the proliferation of bf16. Although it certainly doesn't hurt compared to fp16, in my testing even with A100 GPUs optimized for it, both training speed and inference quality are the same between bf16 and fp16.
Fp16 makes it easy to accidentally overflow, especially around summation operations.
Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]
#6Somehow missed this from WWDC23, but it looks like Sonoma will add support for bfloat16 with Metal, and there's an active PR to add support with the PyTorch MPS back-end (PR #99272). Since M2 added bfloat16 support at the hardware level, I'm assuming this will only be supported on M2 Macs. That maxed out Mac Studio M2 w/ 192GB of memory now looks more appealing...
Visible in the unofficial documentation for AMX instructions too - M2 only bf16 functionality - https://github.com/corsix/amx/blob/main/matfp.md
This matfp instruction computes an outer product and is a kernel for matrix multiplication.
Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]
#7Confirmed Apple M1 lacks bfloat16 support completely -
M1:
hw.optional.arm.FEAT_BF16: 0
vs
M2:
hw.optional.arm.FEAT_BF16: 1
Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]
#8I'm still confused by the proliferation of bf16. Although it certainly doesn't hurt compared to fp16, in my testing even with A100 GPUs optimized for it, both training speed and inference quality are the same between bf16 and fp16.
Sometimes during training, fp16 will cause networks that would converge on fp32, to explode to Infs or NaNs with fp16, because of the limited range. bf16 generally speaking fixes that.
It's true also that fp16 is often manageable with enough batch/layer norm and gradient clipping.
Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]
#9I'm still confused by the proliferation of bf16. Although it certainly doesn't hurt compared to fp16, in my testing even with A100 GPUs optimized for it, both training speed and inference quality are the same between bf16 and fp16.
My understanding is for certain types of networks BF16 will train better than FP16, given the additional protection against exploding gradients and loss functions with the extended range of BF16 - at the loss of precision.
Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]
#10I'm still confused by the proliferation of bf16. Although it certainly doesn't hurt compared to fp16, in my testing even with A100 GPUs optimized for it, both training speed and inference quality are the same between bf16 and fp16.
(Not an ML guy.) bf16 and fp16 should be comparable if the weights are of the same magnitude, but what happens in a network where the weights are poorly regularized?