Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bfloat16 support coming to Apple's Metal and PyTorch [video]

developer.apple.com

Post reply on HN
Print view View original

11–20 of 55 posts

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#11

Post by dlewis1788 »

kzrdude wrote: post #6
eoskx wrote: post #2

Somehow missed this from WWDC23, but it looks like Sonoma will add support for bfloat16 with Metal, and there's an active PR to add support with the PyTorch MPS back-end (PR #99272). Since M2 added bfloat16 support at the hardware level, I'm assuming this will only be supported on M2 Macs. That maxed out Mac Studio M2 w/ 192GB of memory now looks more appealing...

Visible in the unofficial documentation for AMX instructions too - M2 only bf16 functionality - https://github.com/corsix/amx/blob/main/matfp.md This matfp instruction computes an outer product and is a kernel for matrix multiplication.

I didn't even know about Apple's AMX instructions until I clicked on your link. Very interesting - thanks!

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#12

Post by eoskx »

bobbylarrybobby wrote: post #10
minimaxir wrote: post #3

I'm still confused by the proliferation of bf16. Although it certainly doesn't hurt compared to fp16, in my testing even with A100 GPUs optimized for it, both training speed and inference quality are the same between bf16 and fp16.

(Not an ML guy.) bf16 and fp16 should be comparable if the weights are of the same magnitude, but what happens in a network where the weights are poorly regularized?

Someone commented below that with enough batchnorm/layernorm/etc. and/or gradient clipping you can manage it, but BF16 just makes life easier if you can live without some precision.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#13

Post by londons_explore »

dlewis1788 wrote: post #7

Confirmed Apple M1 lacks bfloat16 support completely - M1: hw.optional.arm.FEAT_BF16: 0 vs M2: hw.optional.arm.FEAT_BF16: 1

Luckily BF16 is just a truncated FP32. That means that the hardware can do BF16, just you don't get any performance benefit compared to FP32 (and depending on the hardware design, you might also have to space the data 4 bytes apart rather than 2), so you lose the memory bandwidth and RAM usage benefits too.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#15

Post by londons_explore »

It's a shame that large language models are mostly moving to 4 bit weights for inference, and a bunch of papers have shown promising techniques for training in 4 bit too...

Remember that switching from 16 bit to 4 bit lets you have 4x as many weights, 4x as many weights loaded from RAM per second, and ~1/16 of the silicon area for the calculations (a multiplier scales with approximately the number of bits squared). That smaller silicon area will let you do more per $ too...

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#16

Post by pklausler »

londons_explore wrote: post #13
dlewis1788 wrote: post #7

Confirmed Apple M1 lacks bfloat16 support completely - M1: hw.optional.arm.FEAT_BF16: 0 vs M2: hw.optional.arm.FEAT_BF16: 1

Luckily BF16 is just a truncated FP32. That means that the hardware can do BF16, just you don't get any performance benefit compared to FP32 (and depending on the hardware design, you might also have to space the data 4 bytes apart rather than 2), so you lose the memory bandwidth and RAM usage benefits too.

Conversions from IEEE-32 to BF16 don't round?

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#17

Post by londons_explore »

pklausler wrote: post #16
londons_explore wrote: post #13

Earlier quoted context omitted.

Luckily BF16 is just a truncated FP32. That means that the hardware can do BF16, just you don't get any performance benefit compared to FP32 (and depending on the hardware design, you might also have to space the data 4 bytes apart rather than 2), so you lose the memory bandwidth and RAM usage benefits too.

Conversions from IEEE-32 to BF16 don't round?

I don't believe the standard defines it. I believe implementations truncate (ie. round towards zero).

Remember BF16 was invented specifically to be able to be backwards compatible with existing silicon - and pulling 2 bytes out of 4 is a far cheaper operation than any rounding.

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#18

Post by isoprophlex »

londons_explore wrote: post #15

It's a shame that large language models are mostly moving to 4 bit weights for inference, and a bunch of papers have shown promising techniques for training in 4 bit too... Remember that switching from 16 bit to 4 bit lets you have 4x as many weights, 4x as many weights loaded from RAM per second, and ~1/16 of the silicon area for the calculations (a multiplier scales with approximately the number of bits squared). T…

What?! Can you also train with quantization? Incredible! I'd have thought the gradients were way too ugly for any convergence with 4 bits.

Any particularly good papers you can recommend me on the topic?

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#19

Post by sbierwagen »

isoprophlex wrote: post #18
londons_explore wrote: post #15

It's a shame that large language models are mostly moving to 4 bit weights for inference, and a bunch of papers have shown promising techniques for training in 4 bit too... Remember that switching from 16 bit to 4 bit lets you have 4x as many weights, 4x as many weights loaded from RAM per second, and ~1/16 of the silicon area for the calculations (a multiplier scales with approximately the number of bits squared). T…

What?! Can you also train with quantization? Incredible! I'd have thought the gradients were way too ugly for any convergence with 4 bits. Any particularly good papers you can recommend me on the topic?

A group at IBM has been working on minifloat training for a while. Here's a paper from 2020 on FP4 training: https://papers.nips.cc/paper/2020/file/13b919438259814cd5be8...

Re: Bfloat16 support coming to Apple's Metal and PyTorch [video]

#20

Post by sillysaurusx »

londons_explore wrote: post #13
dlewis1788 wrote: post #7

Confirmed Apple M1 lacks bfloat16 support completely - M1: hw.optional.arm.FEAT_BF16: 0 vs M2: hw.optional.arm.FEAT_BF16: 1

Luckily BF16 is just a truncated FP32. That means that the hardware can do BF16, just you don't get any performance benefit compared to FP32 (and depending on the hardware design, you might also have to space the data 4 bytes apart rather than 2), so you lose the memory bandwidth and RAM usage benefits too.

At that point it’d be better to do everything in fp32. The hardware can’t do bf16 in the way you’re saying; the conversions would consume all your time.
Post reply on HN